tutor
tutor copied to clipboard
Add support for running multiple Celery queues
Is your feature request related to a problem? Please describe.
Open edX already has implemented solutions for running multiple celery queues such as the old known high
and high_mem
default queues however, tutor operates by using a single worker deployment with a default pool (prefork
) and this is not always as performant as desired as that pool is designed for CPU intensive tasks such as the ones related to grading, while I/O bound tasks such as the ones used in Aspects would benefit from having a gevent
pool which used green threads aka virtual threads to accomplish high levels of concurrency. This has already been tested and implemented in tutor-contrib-celery and the benefits have been notorious as the resources are better handled and I/O bound tasks are resolved pretty fast.
Describe the solution you'd like
Allow tutor users to configure multiple Celery deployments with specialized settings for each queue's tasks. With defaults matching what's currently expected to run on each queue.
Describe alternatives you've considered
Developing a separate plugin for Celery: tutor-contrib-celery however we think the community would benefit from having a standard way to set Celery and also don't need to build a custom openedx images with those requirements: https://github.com/openedx/edx-platform/pull/35591
Additional context
https://celery.school/celery-worker-pools