pylint-airflow icon indicating copy to clipboard operation
pylint-airflow copied to clipboard

Question with C8306 For consistency match the DAG filename with the dag_id

Open louisguitton opened this issue 5 years ago • 1 comments

I personally version my DAGs (when we add tasks or change start_date) using only the dag_id, but leaving the filename.

For example

# dags/mailed_report.py

mailed_report = DAG(
    dag_id='mailed_report_v2',
    default_args=default_args,
    catchup=True,
    template_searchpath=['/usr/local/airflow/sql'],
    schedule_interval='@daily'
)

This triggers C8306. I'm curious what your thoughts are: should the filename be versioned as well? or could the match in https://github.com/BasPH/pylint-airflow/blob/master/src/pylint_airflow/checkers/dag.py#L121 be more flexible?

Here is a quick and dirty way to show you what I mean

dagid_without_version =  f"{dagid}".split("_v")[0]
expected_filename = f"{dagid_without_version}.py" 

louisguitton avatar May 25 '19 11:05 louisguitton

Thanks for pointing this out. I have seen other people version their DAGs too, it would indeed make sense to allow the DAG to start with the filename and have a postfix. Perhaps by checking if the dag_id is contained in the filename or vice versa, to allow for versions/other postfixes in either.

BasPH avatar May 26 '19 07:05 BasPH