pylint-airflow
pylint-airflow copied to clipboard
Question with C8306 For consistency match the DAG filename with the dag_id
I personally version my DAGs (when we add tasks or change start_date) using only the dag_id, but leaving the filename.
For example
# dags/mailed_report.py
mailed_report = DAG(
dag_id='mailed_report_v2',
default_args=default_args,
catchup=True,
template_searchpath=['/usr/local/airflow/sql'],
schedule_interval='@daily'
)
This triggers C8306. I'm curious what your thoughts are: should the filename be versioned as well? or could the match in https://github.com/BasPH/pylint-airflow/blob/master/src/pylint_airflow/checkers/dag.py#L121 be more flexible?
Here is a quick and dirty way to show you what I mean
dagid_without_version = f"{dagid}".split("_v")[0]
expected_filename = f"{dagid_without_version}.py"
Thanks for pointing this out. I have seen other people version their DAGs too, it would indeed make sense to allow the DAG to start with the filename and have a postfix. Perhaps by checking if the dag_id is contained in the filename or vice versa, to allow for versions/other postfixes in either.