Explore and implement task-level memory monitoring and display in UI

[omkar-foss]

Description

Feedback from Airflow Debugging Survey 2024:

• Tools to accurately monitor memory usage on a task-by-task basis.

Use case/motivation

Goals for this issue are the following:

• Explore feasibility of monitoring memory usage at task-level.
• Display the memory usage information in Airflow UI.

Related issues

Parent Issue: https://github.com/apache/airflow/issues/40975

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct