airflow-chart icon indicating copy to clipboard operation
airflow-chart copied to clipboard

Published 20 hours ago • verified publisher icon astronomer

How to update DAGs without building a new Airflow Docker image?

Open Minyus opened this issue 4 years ago • 2 comments

According to the README.md,

The recommended way to update your DAGs with this chart is to build a new docker image with the latest code

Would it be possible to update DAGs without building a new Airflow Docker image?

Minyus avatar Jun 24 '20 16:06 Minyus

Hey @Minyus, this is the approach we've taken since we need to also allow all of our customers to update os packages and python packages, in addition to the DAGs.

This chart does not have support for it yet, but there are some folks discussing it in the official apache project. It could absolutely be done and we'd need to support the traditional git fetch, or rsync style of DAG deployment.

How would you like to ideally be able to deploy your DAGs?

schnie avatar Jun 24 '20 16:06 schnie

Thanks for your reply, @schnie . Noted. The following options would be convenient.

  • Upload a zip file including .py files in the Airflow web UI
  • Upload a .py file in the Airflow web UI
  • HTTP request to post the Python code to the server
  • rsync -r to a directory in the server

Minyus avatar Jun 24 '20 17:06 Minyus

We have handled this with git-sync, nfs, and are about to release a dag-only deploy feature. https://docs.astronomer.io/software/deploy-cli

danielhoherd avatar Feb 08 '24 01:02 danielhoherd