cloudtasker icon indicating copy to clipboard operation
cloudtasker copied to clipboard

Fix issue when :until_executed lock prevents cron job from scheduling next job

Open donnguyen opened this issue 2 years ago • 0 comments

Hi alachaum, thank you for the great work again.

We just found a rare case when using both cron job and unique job (:until_executed in specific).

Basically, we have a job with cloudtasker_options lock: :until_executed and the job is scheduled by cron job feature. The issue could happen like below:

  1. When it first runs, because of no lock (I don't know this should be the case? Perhaps a separate thing need to look into?), it schedules next job successfully, now the lock is created.
  2. Now the job will be processed within Unique server middleware, because of the lock just has been created in step 1, the actual job will be rejected/bypassed (we are using default reject conflict strategy, but the result is still Job done. The lock isn't cleared <- This is an issue.
  3. Next time it runs, when it tries to schedule next run, an error will be raised here because of the lock in the step 2. -> Job fails.

So the solution we come up with is that:

  1. Cron job server middleware always need to run before Unique job -> this to make sure no lock left when we try to schedule next run.
  2. Only schedule next run in cron job if the job has been processed successfully and lock has been cleared.

Any thoughts?

donnguyen avatar Aug 04 '22 14:08 donnguyen