Restart UI, Scheduler, Workers on detecting new DAGs in the DAG folder

[barney-s]

1. Drain celery workers nodes and k8s executor pods to prepare for restart
2. Restart airflow UI and Scheduler on detecting new DAGs in the DAG folder

[pacuna]

Hi @barney-s , so right now what's the proper way of adding a new dag to the cluster?

[barney-s]

There isnt unfortunately. Thinking restarting the workers would pull the latest. But not a clean solution.

[barney-s]

@pacuna Did the config dag_dir_list_interval not help you ? I believe thats the canonical way airflow supports refreshing DAGs. https://cwiki.apache.org/confluence/display/AIRFLOW/Scheduler+Basics#Configuration

You can set that config using cluster.spec.config.airflow. Set the env AIRFLOW__SCHEDULER__ DAG_DIR_LIST_INTERVAL

By default dags are refreshed every 5 minutes.

To enable continuous sync, use git or gcs dag source with once disabled.