How does TFX support Online learning?

[liutongxuan]

Online training need to run Trainer(and pre-components) every minutes, and incrementally push to serving. Any idea of TFX support such scenario.

[rmothukuru]

@liutongxuan,
Can you please look into this TFX documentation and let us know if it helps. Thanks!

[liutongxuan]

@rmothukuru @zhitaoli it seems like continuous learning (https://www.usenix.org/system/files/opml19papers-baylor.pdf), Our requirement is that samples are from realtime online service and save model need to be generated incrementally based on realtime data every minute.

[liutongxuan]

But we found TFX need to save intermediate results into meta store for each component, seems not good performance in our scenario.

[steven-mi]

You could schedule your pipeline with a orchestrating tool e.g. in airflow you could specify to run the DAG every minute.

[zhitaoli]

You could schedule your pipeline with a orchestrating tool e.g. in airflow you could specify to run the DAG every minute.

I don't think we have seen anymore deploying the system in that style yet, including ourselves. While I don't know anything which could break for sure, a couple of things to watch out:

1. watch growth on both artifact storage (GCS/S3/HDFS) and MLMD. Especially the latter, as the volume of executions and artifacts could even hurt system performance.
2. it's one thing of how faster we schedule the pipeline to run, but another thing how fast it can run itself. Even if airflow could schedule a DAG every minute reliably (which needs its own verification), the cost of scheduling and overhead of node execution would kick in at this level. It's good to understand how that affects your goal.
3. many nodes in a typical TFX/ML pipeline could take long to finish. I think our "async" pipeline design in this RFC might be more friendly. We are still figuring out how to bring this to OSS. Happy to engage more.

[steven-mi]

I think I maybe misunderstood your goal. I was under the assumption that you wanted to run your pipeline multiple times, which is why I suggests to use the integrated orchestrating tool.

As far as I understood your post, I think your main problem is that the execution time is not consistence, because the amount of data is not. I don't believe that async mode is a good idea, since it will give you more headache than needed. If one would introduce asynchronously to the pipeline, then you would have the functionality twice, which also would give you additional overhead. Additionally you have the problem you mentioned in your RFC which cannot be solved by pushing the "best" model.

Keeping that in mind, wouldn't be a reduction in datapoint more optional? You could limit your data to a certain amount to ensure consistency throughout every run.