Actor priority ignored

[mihasajko]

Issues

GitHub issues are for bugs. If you have questions, please ask them on the discussion board.

Checklist

• [x] Does your title concisely summarize the problem?
• [x] Did you include a minimal, reproducible example?
• [x] What OS are you using?
• [x] What version of Dramatiq are you using?
• [x] What did you do?
• [x] What did you expect would happen?
• [x] What happened?

What OS are you using?

macOS 11.6

What version of Dramatiq are you using?

1.11.0

What did you do?

I tried to send a prioritized message in this way:

import dramatiq
import time
import random


@dramatiq.actor(priority=0)
def high_priority_task(n: int):
    print(f"Starting order {n}")
    time.sleep(5)
    print(f"Ending order {n}")

@dramatiq.actor(priority=10)
def low_priority_task(n: int):
    print(f"Starting order {n}")
    time.sleep(5)
    print(f"Ending order {n}")

if __name__ == "__main__":
    for n in range(20):
        if n == 10:
            high_priority_task.send(n)
        else:
            low_priority_task.send(n)

Broker: RabbitMQ

To ensure a bottleneck, I started a worker with 1 process and 1 thread.

What did you expect would happen?

I expected that order 10 would be prioritized and processed earlier.

What happened?

All orders get processed sequentially.

[mparent61]

I ran into this problem with the Redis broker, as the priorities only apply to pre-fetched tasks pulled down on an individual worker. There might be a way to get this to work with RabbitMQ - https://github.com/Bogdanp/dramatiq/issues/412.

If anybody were to make a "global" priority work with Redis I'd love to know about it! Our current solution is to have separate queues, "default" and "high_priority" configured per-actor.

[kastman]

Looks like Celery provides this with a redis backend by splitting multiple queues: see a SO question and the redis docs. There's also an implementation using sorted sets which could possibly be used.

I unfortunately don't think I have time to set up a "global" priority queue, but would you welcome a doc change that says priority is only implemented in the rabbitmq broker right now?

[mparent61]

@kastman - Yes, a doc change would be quite helpful (perhaps pointing to separate queues as a Redis workaround), as I spent a bit of time determining that Redis backend didn't actually support global priority (and I'm sure others have/will too). I understand developer time is limited, and am grateful for this project!