django-q2 icon indicating copy to clipboard operation
django-q2 copied to clipboard

Is it safe to schedule tasks from tasks and then wait for the results?

Open elnygren opened this issue 8 months ago • 0 comments

If the cluster is full of tasks that schedule other tasks, and wait for the results, can it lead to a situation where all workers have a task that is waiting on other tasks i.e a deadlock?

Practical example:


def subtasks():
  """task that creates an iterable and waits for the results""""
  r = async_iter(subtask, ...)
  r.results(wait=...)

def main_task()
  c = Chain()
  c.append(subtask, ...)
  c.append(something else)
  c.append(send_email, "subtasks are completed!")
  c.run()

Is it possible that the cluster has more main_tasks than workers and each worker is stuck with a main_task that is just waiting and waiting for the subtasks that never get scheduled?

ps. What would be the correct way of implementing a Chain of tasks where some of the tasks require doing parallel work such as downloading images? Ofc. I can just chain everything in a single chain but then there is no parallelism. I can also use Iter but then there might be too much parallelism, eg. if I want to batch several Iters in a Chain.

elnygren avatar Jun 13 '24 12:06 elnygren