kedro-plugins icon indicating copy to clipboard operation
kedro-plugins copied to clipboard

kedro-airflow: Allow passing `conf_source` to the created DAG

Open ankatiyar opened this issue 9 months ago • 2 comments

Description

Add --conf-source flag to kedro airflow create which will pass this on to the Jinja template used to generate the DAGs airflow_dag_template.j2

Context

To run a Kedro project as a DAG on airflow currently -

  • You should package and install your kedro project in the executor env
  • Copy the config from the Kedro project to the Airflow project folder https://docs.kedro.org/en/stable/deployment/airflow.html#create-prepare-and-package-example-kedro-project
  • If something changes in the conf, one must also remember to copy over the changes

Additionally,

  • KedroSession.create() now allows passing conf_source path
  • kedro package creates a .whl file for the project pipelines etc and a .tar.gz file for the conf/ folder of the project which can be saved elsewhere

Possible Implementation

  • Add --conf-source to the CLI command which passes this info the Jinja template airflow_dag_template.j2

Possible Alternatives

(Optional) Describe any alternative solutions or features you've considered.

ankatiyar avatar May 08 '24 15:05 ankatiyar