Support of custom celery configs

[arorasachin9]

Description

Currently airflow supports few celery configs only like worker concurrency, worker prefetch multiplier etc. All others config are not supported. One of main out of them is worker max tasks per child which helps in recycling the celery worker processes. It helps in reclaiming the resources of the processes which got consumed due to high resource utilising task.

Use case/motivation

We have a task which consumes very high memory. As task keeps on changing the processes so our memory keeps on increasing. To recycle the process after certain number of task is the celery support but currently not supported in airflow.

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.