Automate model training with Airflow?

[everglory99]

Hi, We have some new models written under lore framework and we typically use Airflow to automate most of our existing data and ML pipelines. Thus, we would love to automate the lore training pipeline in Airflow as well. Before implementing this, I am wondering how you guys at Instacart manage the ML training pipelines in production? Thanks for the help!

[montanalow]

I think airflow makes sense, currently we have a few methods depending on the team and needs of the model.

1. cron jobs
2. airflow
3. aws batch
4. domino data labs
5. custom ETL