How to use a group periodically - missing documentation
- [x] I have checked the issues list for similar or identical bug reports.
- [x] I have checked the pull requests list for existing proposed fixes.
- [x] I have checked the commit log to find out if the bug was already fixed in the main branch.
- [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).
Related Issues and Possible Duplicates
Related Issues
- None
Possible Duplicates
- None
Description
In the documentation, there is indication on how to add a single periodic task but I didn't find any indication on how to schedule a periodic group.
Doing sender.add_periodic_task(10, my_group, name='group task name') raises an exception.
So I ended up creating one main task that spawns many subtasks, which is discouraged in the docs:
job = group([subtask.s(item) for item in range(10)])
result = job.apply_async()
ret_val = result.get(disable_sync_subtasks=False)
Suggestions
Provide information on how to schedule a group job periodically.
Where in documentation is that indication?
https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html
This shows how to add a single periodic task. I'm looking for documentation on how to schedule a periodic group.
You can't. Groups/Chords/Chains and others are part of what is called workflows of tasks [0]. beat schedules are designed to initiate a individual task as far as I know.
It's very much possible to use initiating tasks to kick start your workflows:
@task
def op_workflow_a():
from celery import group
from my_app.tasks import send_email
emails = ["[email protected]", "[email protected]"]
group_task = group(send_email.s(email=email) for email in emails)
group_result = group_task.apply_async()
return group_result.id
[0] https://docs.celeryq.dev/en/stable/userguide/canvas.html#groups
That is already what I'm doing. However, after the group_result = group_task.apply_async() line, I have a group_result.get(disable_sync_subtasks=False) because disable_sync_subtasks is necessary here. What happens if I do not await for completion? Is there a risk one of the subtasks is interrupted/killed depending on the circumstances?