work icon indicating copy to clipboard operation
work copied to clipboard

Ability to retry dead jobs manually

Open CodechCFA opened this issue 3 years ago • 3 comments

One of the biggest reasons we used https://github.com/gocraft/work is that it has the ability in the UI to retry jobs manually that have exhausted their automatic retries. Given that this project has moved the UI solution to prometheus/grafana, are there any plans for covering the need to retry dead jobs manually?

CodechCFA avatar Mar 10 '21 17:03 CodechCFA

The default behavior changes to retrying job forever for simplicity, but here are some ways to customize this:

  1. use work.ErrDoNotRetry. This will cause a job to be not acked, and not reenqueued. You can then use your middleware to enqueue with your preferred backoff period.
  2. use discard.AfterXXX middleware. It means this job will be automatically removed after a certain period, or retry count.

taylorchu avatar Mar 11 '21 15:03 taylorchu

That makes sense for managing the number of times a job retries. My question was around what to do with a job after it has exhausted its retries. Being able to manage dead jobs is incredibly important because it allows you to investigate what jobs failed, why they failed, and what to do with them, such as retrying them again or deleting them. https://github.com/gocraft/work Allows exactly this. Without having that ability here, we'll have to continue using https://github.com/gocraft/work

CodechCFA avatar Mar 11 '21 16:03 CodechCFA

I see. (in v2, there is no dead job because there is no concept of max retries or exhausted max retries; the job just keeps retrying forever. )

This will be a nice addition with a middleware package to put jobs with exhausted max retries into a different place, and provide UI to discard or retry (put them back to the normal queue).

taylorchu avatar Mar 11 '21 17:03 taylorchu