airflow-declarative icon indicating copy to clipboard operation
airflow-declarative copied to clipboard

Allow to disable imports on scheduler side

Open kxepal opened this issue 7 years ago • 0 comments

Case: we have a scheduler service which runs under some specific Python version. Say 2.7. Now we need to run DAGs for Python 3.x projects. Since scheduler does class/callback import during schema processing it's not possible to do without setup another scheduler service specially for Python 3.x. Another scheduler process means that we need another AIRFLOW_HOME directory to separate files and another configuration and one more critical for production service to maintain.

Theoretically, we can not do any imports on scheduler side since all the important information for DAG run creation is a plain Python data. In this case we would have single scheduler process, single Airflow configuration, single place for DAGs files and two groups of workers per each Python version.

This also should speedup scheduling alot since it woulnd't spend a lot of time diving into imports rabbit hole. Some of them may take a lot of time and not all of them are without side effects (hi, theano!).

kxepal avatar Jul 30 '17 11:07 kxepal