kedro-plugins
kedro-plugins copied to clipboard
kedro-airflow: Allow passing `conf_source` to the created DAG
Description
Add --conf-source
flag to kedro airflow create
which will pass this on to the Jinja template used to generate the DAGs airflow_dag_template.j2
Context
To run a Kedro project as a DAG on airflow currently -
- You should package and install your kedro project in the executor env
- Copy the config from the Kedro project to the Airflow project folder https://docs.kedro.org/en/stable/deployment/airflow.html#create-prepare-and-package-example-kedro-project
- If something changes in the conf, one must also remember to copy over the changes
Additionally,
-
KedroSession.create()
now allows passingconf_source
path -
kedro package
creates a.whl
file for the project pipelines etc and a.tar.gz
file for theconf/
folder of the project which can be saved elsewhere
Possible Implementation
- Add
--conf-source
to the CLI command which passes this info the Jinja templateairflow_dag_template.j2
Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.