AIP-44 Migrate TaskInstance.schedule_downstream_tasks to Internal API

Open mhenc opened this issue 2 years ago • 3 comments

https://github.com/apache/airflow/blob/main/airflow/models/taskinstance.py#L2570

used in

https://github.com/apache/airflow/blob/main/airflow/jobs/local_task_job.py#L220

Feb 02 '23 12:02 mhenc

I can work on this one

Feb 14 '23 16:02 vincbeck

Yep. But let's agree if the proposal I added here makes sense https://github.com/apache/airflow/pull/29513#discussion_r1106553896 first. - this will likely change the approach for that one.

Feb 15 '23 01:02 potiuk

From AIP-44 descriptions I understood that this "mini scheduler" might be better pushed-back to scheduler as a lot of DB access is needed. Has opinion changed sind AIP write-up? Else I'd propose to "just skip" the schedule after task execution like if AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION=False is set.

THat would be very simple as a one-line addition in airflow/jobs/local_task_job_runner.py:243

May 19 '24 20:05 jscheffl

Yes. I think it should be done like that AIP-44 goal is not to improve performance but increase isolation, so removing of mini-scheduler is likely a good idea.

May 21 '24 20:05 potiuk

This it too complex to be completed in the "experimental" phase. For now AIP-44 will have "mini-scheduler" disabled.

Jul 27 '24 18:07 potiuk