rq
rq copied to clipboard
rq
Gevent, Concurrency and External Methods
Hi all.
Currently looking for an alternative to Celery + RabbitMQ (want to change because of the high complexity and maintenance costs). Most of my tasks are I/O bound, so having concurrency with something like Gevent would be great. I've seen a few issues on the subject starting all the way back in 2012, but just can't find something "official" in the documentation. I wonder if this hasn't (won't) be implemented, or maybe I just missed it.
Depending on the application running too many workers is not optimal considering the linear increase in RAM usage, so if was the case of not having concurrency, I wonder if it would be possible to call an external method from the main application. This is the only way I can imagine of avoiding tons of memory usage and low performance on the task execution. In my current use case, my app uses about ~250MB of memory at startup but I could easily spin off some tasks to a 25MB container. As far as I've seen so far, the API references the function itself (and not a string like arq does for example), so I'm not sure if this is something that has been solved in RQ even though the discussions on the topic are fairly old.
I maybe be wrong though, I just started to assess the library, so please let me know if I'm missing something. Thanks a lot
We haven't gotten around to implementing concurrent workers, unfortunately. This is something I'm keen to add, but never got around to it since it's going to be quite a big architectural change.
@selwin thanks for the response, I missed the notification, so sorry for the delayer answer.
Do you think it would be possible to add an extra GeventWorker class?
While writing this I actually just found this repo. Have you guys considered adding something similar?
For future reference, a first test is linked here https://github.com/rq/rq/pull/885