django-q2
django-q2 copied to clipboard
Is it safe to schedule tasks from tasks and then wait for the results?
If the cluster is full of tasks that schedule other tasks, and wait for the results, can it lead to a situation where all workers have a task that is waiting on other tasks i.e a deadlock?
Practical example:
def subtasks():
"""task that creates an iterable and waits for the results""""
r = async_iter(subtask, ...)
r.results(wait=...)
def main_task()
c = Chain()
c.append(subtask, ...)
c.append(something else)
c.append(send_email, "subtasks are completed!")
c.run()
Is it possible that the cluster has more main_task
s than workers and each worker is stuck with a main_task
that is just waiting and waiting for the subtasks that never get scheduled?
ps. What would be the correct way of implementing a Chain of tasks where some of the tasks require doing parallel work such as downloading images? Ofc. I can just chain everything in a single chain but then there is no parallelism. I can also use Iter but then there might be too much parallelism, eg. if I want to batch several Iters in a Chain.