celery-redis-priority-test
celery-redis-priority-test copied to clipboard
Used to determine how priorities are working in celery with a redis backend
celery-redis-priority-test
Used to determine how priorities are working in celery with a redis backend
run in docker
docker-compose up
How to do the thing:
run celery in one shell
celery worker -A tasks -Q a-high,b-medium,c-low -Ofair -c1 --prefetch-multiplier=1
and run tests in another
green
(i dont have pytest commented yet. will do soon)
https://medium.com/@hengfeng/how-to-route-tasks-in-celery-rabbitmq-e0e22c178e45
Undocumented, or incorrect assertions of Celery and Redis:
1: task_inherit_parent_priority
is available for the Redis backend, and not just AMQP
2: Task priority is low to high, eg: tasks with a priority of 0
are handeled before 9
3: If you set the broker_transport_options
to {"queue_order_strategy": "sorted"}
, it will prioritize tasks from queues in alphabetical order
4: Tasks are ran in order of sorting first by task priority, and second by queue order
5: If you override task priority by setting broker_transport_options
to {"priority_steps":[0, 2, 4, 6, 8]}
, 0 will be the highest priority. If you invert that to {"priority_steps":[8, 6, 4, 2, 0]}
, 8 will be the highest. The order of priority_steps
dictates priority order.