Use pubsub and blocking for improved response times

[m-kuhn]

arq currently uses a polling mode (default 0.5 seconds) to fetch jobs on worker side and also to fetch results. I think it should be possible to use blocking requests (on worker side) and pubsub (possibly with keyspace notifications, but not necessarily) for fetching the result.

This should be quite reliable in most scenarios, however it can occur that a job gets lost (there is no guarantee for pubsub delivery after all), so one could still poll from time to time to be on the safe side.

Are there any opinions on development in this direction?

From https://github.com/samuelcolvin/arq/pull/374#issuecomment-1570580555

[m-kuhn]

CC @arturzx

[gg718]

See also how rq recently implemented this: https://github.com/rq/rq/pull/1939

[JonasKs]

A draft PR welcome.😊

[samuelcolvin]

My plan is to move to redis streams which are by far the best solution for tasks like those performed by arq. Stay tuned for a big update.

[samuelcolvin]

See #437.

[Wh1isper]

I agree that redis stream is the best option right now. Can't wait for the arq update, I developed a redis stream based library, aka brq. If you're in a hurry, you may try it now.