airflow-declarative
airflow-declarative copied to clipboard
Allow to disable imports on scheduler side
Case: we have a scheduler service which runs under some specific Python version. Say 2.7. Now we need to run DAGs for Python 3.x projects. Since scheduler does class/callback import during schema processing it's not possible to do without setup another scheduler service specially for Python 3.x. Another scheduler process means that we need another AIRFLOW_HOME directory to separate files and another configuration and one more critical for production service to maintain.
Theoretically, we can not do any imports on scheduler side since all the important information for DAG run creation is a plain Python data. In this case we would have single scheduler process, single Airflow configuration, single place for DAGs files and two groups of workers per each Python version.
This also should speedup scheduling alot since it woulnd't spend a lot of time diving into imports rabbit hole. Some of them may take a lot of time and not all of them are without side effects (hi, theano!).