apache
Metric scheduler.tasks.running is never updated from 0
Apache Airflow version
2.2.3 (latest released)
What happened
The metric scheduler.tasks.running is populated from the variable num_tasks_in_executor in scheduler_job.py. However, num_tasks_in_executor is initialized to 0 and then never updated again.
Function _executable_task_instances_to_queued:
https://github.com/apache/airflow/blob/c59001d79facf7e472e0581ac8a538c25eebfda7/airflow/jobs/scheduler_job.py#L328
https://github.com/apache/airflow/blob/c59001d79facf7e472e0581ac8a538c25eebfda7/airflow/jobs/scheduler_job.py#L457
I considered just opening a PR modifying that value with the rest of these "counter" modifications: https://github.com/apache/airflow/blob/c59001d79facf7e472e0581ac8a538c25eebfda7/airflow/jobs/scheduler_job.py#L449-L452
However, I'm not sure if this is the best course of action because we don't know if the tasks is successfully added until the the list of executable TIs is returned within _critical_section_execute_task_instances, and those TIs are attempted to be queued in _enqueue_task_instances_with_queued_state.
https://github.com/apache/airflow/blob/c59001d79facf7e472e0581ac8a538c25eebfda7/airflow/jobs/scheduler_job.py#L534-L540
https://github.com/apache/airflow/blob/c59001d79facf7e472e0581ac8a538c25eebfda7/airflow/executors/base_executor.py#L73-L85
What you expected to happen
No response
How to reproduce
No response
Operating System
Ubuntu 19.10
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Yeah. Confirmed it does look suspiciously 0-always metrics :), Would you maube like to attempt to fix that one @easontm ?
Opening PR and discussing there, might be the best way to get to the right solution as it will drag attention of those who are closer to the code.
I won't be able to do it right away because of other obligations, but I'll see what I can do in a few weeks!
I believe we could close this out as we removed that metrics https://github.com/apache/airflow/pull/30374. You should be using executor running and executor queued task metrics. cc: @vincbeck
