celery icon indicating copy to clipboard operation
celery copied to clipboard

How to use a group periodically - missing documentation

Open jonasN5 opened this issue 2 years ago • 4 comments

  • [x] I have checked the issues list for similar or identical bug reports.
  • [x] I have checked the pull requests list for existing proposed fixes.
  • [x] I have checked the commit log to find out if the bug was already fixed in the main branch.
  • [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Description

In the documentation, there is indication on how to add a single periodic task but I didn't find any indication on how to schedule a periodic group. Doing sender.add_periodic_task(10, my_group, name='group task name') raises an exception. So I ended up creating one main task that spawns many subtasks, which is discouraged in the docs:

job = group([subtask.s(item) for item in range(10)])
result = job.apply_async()
ret_val = result.get(disable_sync_subtasks=False)

Suggestions

Provide information on how to schedule a group job periodically.

jonasN5 avatar Mar 13 '24 07:03 jonasN5

Where in documentation is that indication?

vesnikos avatar Apr 21 '24 14:04 vesnikos

https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html

This shows how to add a single periodic task. I'm looking for documentation on how to schedule a periodic group.

jonasN5 avatar Apr 21 '24 14:04 jonasN5

You can't. Groups/Chords/Chains and others are part of what is called workflows of tasks [0]. beat schedules are designed to initiate a individual task as far as I know.

It's very much possible to use initiating tasks to kick start your workflows:


@task
def op_workflow_a():
  from celery import group
  from my_app.tasks import send_email
  emails = ["[email protected]", "[email protected]"]
  group_task = group(send_email.s(email=email) for email in emails)
  group_result = group_task.apply_async()
  
  return group_result.id

[0] https://docs.celeryq.dev/en/stable/userguide/canvas.html#groups

vesnikos avatar Apr 21 '24 14:04 vesnikos

That is already what I'm doing. However, after the group_result = group_task.apply_async() line, I have a group_result.get(disable_sync_subtasks=False) because disable_sync_subtasks is necessary here. What happens if I do not await for completion? Is there a risk one of the subtasks is interrupted/killed depending on the circumstances?

jonasN5 avatar May 06 '24 09:05 jonasN5