[Metrics&Observability] Track task manager metrics

[kdelee]

ISSUE TYPE

• Feature Idea

SUMMARY

Track timing and stats of the task manager. This can include:

• Number of jobs collected by task manager (by type)
• Number of jobs scheduled in a run of the task manager
• Number of jobs blocked by "soft" dependencies like allow_simultaneous=False
• Time it took to run different stages of the task manager (e.g. _schedule, process_pending_tasks, etc)

@fosterseth has done some work for efficient collection of stats in subsystem metrics. Since that's how we measure the callback receiver, I think we should use the same tool to measure the task manager timing breakdown.

[kdelee]

https://github.com/ansible/awx/pull/12235

[kdelee]

This is now merged, we need to determine if we want to add any tests

Note: may undergo some updates with work on the task_manager_refactor branch cc @fosterseth @AlanCoding

[obaranov]

Verified and added downstream tests: https://github.com/ansible/tower-qa/pull/7665