airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Fix airflow tasks run --local when dags_folder differs from that of processor

Open dstandish opened this issue 3 years ago • 1 comments

Currently, if your dags_folder differs from that used by the dag processor, then task_run --local will fail to find the dag, because it uses the fileloc value stored in the serialized dag.

To resolve this issue, I add the dags_folder as configured in the dag processor (or whatever process does the serializing) and this allows us to correctly determine the relative path even when the current dags folder is different.

~depends on https://github.com/apache/airflow/pull/26536 getting merged first~

dstandish avatar Sep 19 '22 22:09 dstandish

Needs reabase :). But looks good. @mhenc?

potiuk avatar Sep 22 '22 09:09 potiuk

OK so... i made it so this is just a private attr on ser dag object...

So really this just fixes the existing feature without adding any new backcompat promises.

And it is called dags folder not processor subdir, but, as has been established, the two are not the same thing, and the present attr is always going to tell the truth... so... @potiuk @mhenc does it get your blessing?

dstandish avatar Sep 27 '22 22:09 dstandish

Yes, looks good for me. Thank you!

mhenc avatar Sep 28 '22 07:09 mhenc

Makes sense - one static check to fix but good to go :)

potiuk avatar Sep 28 '22 09:09 potiuk