Triggered tasks from web-ui are killed by scheduler because they are orphaned task instances

[nclaeys]

Apache Airflow version

2.2.2

What happened

I have a dag which is scheduled every day and has depends_on_past set to true. When a task fails for a given date, which is expected as the required data is not there. If I want to manually run the next task (for the following day) the run always fails.

The reason for this is that airflow creates next task instances for the following days and sets their state to None as they cannot be scheduled because the previous task instance is in failed state, this is correct and expected behavior. If I now manually trigger the task_run from the web-ui, the task_instance data in the database is not updated and as a consequence the queued_by_job_id is not filled in.

Every 5 minutes the airflow scheduler queries for orphaned tasks, and since my manual run does not have queued_by_job_id filled in, it always gets killed as the scheduler thinks it is orphaned. The scheduler shows the following logs: airflow-scheduler [2022-01-20 11:15:41,784] {scheduler_job.py:1178} INFO - Reset the following 1 orphaned TaskInstances: airflow-scheduler <TaskInstance: testmanual.sample scheduled__2022-01-16T00:00:00+00:00 [running]>

What you expected to happen

I expect that the manual run will not be killed by the scheduler as it thinks it is orphaned and thus that my tasks can still succeed.

If this is expected behavior, It is best to show an error to the user when it tries to run the request stating: Running this task manually is not supported because... Then at least it is clear to the user, now the actual reason is really hidden and not obvious for most users I assume.

How to reproduce

• Create a dag that fails for the start_date and for the following dates just sleeps for 5 minutes and set the start date to yesterday.
• Enable the dag and the run for yesterday will fail (expected)
• Manually trigger the dag for today through the web-ui
• The run for today will be killed by the scheduler every time

Operating System

debian buster

Versions of Apache Airflow Providers

/

Deployment

Other Docker-based deployment

Deployment details

We run airflow on kubernetes and thus use the kubernetes_executor to schedule tasks.

Anything else

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct