Periodic task lifecycle management

[LucasRoesler]

trafficstars

For various reasons, I was looking for a way to "delete a periodic task" which caused me to look into what happens when a period task is scheduled. This mostly lead me to the periodic.py

If I read the implementation correctly, once the function is removed, or at the minimum the decorator is removed from the function, it will no longer be put into the periodic registry when the app/worker starts. This will prevent it from scheduled new instances of that task, but happens to previously scheduled job, for example, if I have a daily task and I remove it, based on my current understanding, it seems like there will be a stray Job left over and one more execution will happen tomorrow. If the function is removed, the worst case is that the Job fails quickly.

It also seems like this might occur if you modify the cron schedule for a task. This one seems like it would be a potential bug.

I was wondering what is the recommended way to handle changes to a periodic task? Have a overlooked something?

[medihack]

Hi @LucasRoesler. You are correct in your assumptions. There is no automatic cancellation of jobs when a task or periodic task is removed. But it should be easy to write a script that looks for jobs of a specific (deleted) task and cancel all those jobs.

[LucasRoesler]

Ok, glad I didn't misread it. I guess it would be as simple as (in pseudo-code)

deprecated_tasks = ["...", ]
for name in deprecated_tasks:
   jobs = app.job_manager.list_jobs(task=name)
   for j in jobs:
      app.job_manager.cancel_job_by_id(job_id=j.id, delete_job=True)

But this would require maintaining a list of deprecated tasks. If I wanted to make this generic, I would probably need to do something like

tasks_in_db = set(app.job_manager.list_tasks())
known_tasks = set(app.tasks.keys())
stale_tasks = tasks_in_db - known_tasks
for name in stale_tasks:
   jobs = app.job_manager.list_jobs(task=name)
   for j in jobs:
      app.job_manager.cancel_job_by_id(job_id=j.id, delete_job=True)

But neither of these really work for the "rescheduling a periodic job" use case, because that task would still be known to the registry. It seems like this is best handled during the periodic defer, it should look for existing unfinished jobs for that tasks, remove it, and then do the rest of the normal defer process.

[onlyann]

@LucasRoesler I am not sure this is how the periodic worker schedules tasks.

In the example of a daily task, it should not be creating a job to start a day in the future but would instead defer the job at the time it should have started (or up to a certain delay).

For instance, say a job is scheduled at 6am daily. The periodic worker would wait until at least 6am to defer a job for that day.

Removing a periodic task should in theory not cause an issue unless a job for that task is deferred and the worker is stopped before the job could be processed.

[LucasRoesler]

Oh yeah, you are totally right and I definitely read through that and I recall noting it, I don't know why that other idea got stuck in my head. Thanks for clarifying it 🙏

[ewjoachim]

Yep, what @onlyann said.

Removing a periodic task should in theory not cause an issue unless a job for that task is deferred and the worker is stopped before the job could be processed.

Another possibility is if you have a very long backlog. One task might be deferred at 6pm but actually executed hours (or days or more) later, so if by this time the workers have been redeployed and don't know about the task, they will error

If the function is removed, the worst case is that the Job fails quickly.

Yeah, you'll get a TaskNotFound