RealOrangeOne
Running async tasks in parallel
Really love this package! Can't wait to see it land in Django core
I was wondering if it was possible to run tasks in parallel?
I have some slow network calls which I call asynchronously, but tasks are still executed sequentially. Is it possible to start processing the next task while awaiting a call in a previous one?
Currently no - each process runs just a single task concurrently. If you need to run multiple tasks at once, you'll need to run multiple worker processes.
It's something I think it would be great to support, particularly for async tasks, but working out the behaviour, especially when relating to non-async tasks is complex.
See also this discussion: https://github.com/RealOrangeOne/django-tasks/discussions/117
[...] especially when relating to non-async tasks is complex.
Can you give a brief explanation or pointer what you mean with that? Having a sync task in the mix would be blocking other tasks, even if all other tasks are async, but that could also happen if all tasks are completely async. Also, I'm not sure if there is a clean way to just use threads instead of an event loop. Anything that adds some concurrency without spawning full processes (and exploding memory requirements) sounds fine to me.
There are certain features which may require child processes for tasks (eg timeouts for sync tasks). So the only way to achieve proper concurrency is to do process-based. The more complexity in the worker process, the harder it is to reason about what it's doing.
I could see an argument for an async-only worker command, which only handles async tasks, and thus has massively improved throughput. But that's a future aspiration.