Clarifying repeating/canceling jobs

[jmonster]

Description

I had a difficult time figuring out the proper incantation to create and destroy repeatable jobs. I'm opening this issue to provide a working example for anyone else who might be googling the issues I ran into.

I ran into the following scenarios as I trial-and-errored my way to this solution.

• a repeating job didn't fire until the repetition time (rather than immediately)
• the job couldn't be deleted
• only 1 job was being created (despite attempting to create 100 jobs with different data payloads)
• jobs weren't being removed when they were done or failed

The core point of confusion seemed to be where/how to pass jobId; it needs to be passed as part of the repeatOpts object.

Correct example

const ONE_SECOND = 1000;
const ARBITRARY_INTERVAL = 3000;

const Queue = require('bull');
const myQueue = new Queue('my-queue-name',
  {
    limiter: { max: 1, duration: ONE_SECOND }, // 1 job/second
    defaultJobOptions: {
      removeOnFail: true,
      removeOnComplete: true
    }
  }
);

const someCollection = [1,2,3];
someCollection.forEach((jobId) => {
  const repeatOpts = { jobId, every: ARBITRARY_INTERVAL }; // note that `jobId` is part of the `repeat` options

  // adding to the queue
  myQueue.add({ arbitraryDataKey: 123 }, { repeat: repeatOpts }); // an object with a `repeat` key

  // removing from the queue
  myQueue.removeRepeatable(repeatOpts); // the repeatOpts are passed directly; not an object with a `repeat` key
});

[manast]

Maybe it would be easier if you explained what are you trying to achieve, and then we can give you tips on how to implement it properly (if possible) with bull.

[jmonster]

Sure, I would love any advice you may have to share

• a set of jobs run on an interval, say daily
• run as soon as I add them to the queue
  • I can't get that to work and have resorted to scheduling two jobs; one for now and one to repeat every 8.64e+7 milliseconds
• If that job were to error/fail in an unrecoverable way (e.g. fetching a particular url returns a 404) I'd like for the job to remove it's repeating self from the queue so that it doesn't continue to try/fail every day
• update the repeating schedule to be more-or-less frequent (e.g. hourly vs daily) depending on some external heuristic (e.g. popularity)
• make <= 1 request per second to stay within the upstream server's rate limit
• avoid accumulating completed/failed jobs for the sake of limiting memory consumption

thanks!

[manast]

Please see my answers below.

• a set of jobs run on an interval, say daily

• run as soon as I add them to the queue

  • I can't get that to work and have resorted to scheduling two jobs; one for now and one to repeat every 8.64e+7 milliseconds

This could probably achieved by generating a cron expression dynamically, however I understand your need, and it could probably implemented as a new feature in bull.

• If that job were to error/fail in an unrecoverable way (e.g. fetching a particular url returns a 404) I'd like for the job to remove it's repeating self from the queue so that it doesn't continue to try/fail every day

That should be possible to do by listening to the failed event and deleting the repeatable job in that case.

• update the repeating schedule to be more-or-less frequent (e.g. hourly vs daily) depending on some external heuristic (e.g. popularity)

Can only be achieved currently by deleting the job and recreating it. Probably also a nice feature to add to bull.

• make <= 1 request per second to stay within the upstream server's rate limit

Rate limiting should already allow you to do that.

• avoid accumulating completed/failed jobs for the sake of limiting memory consumption

You can use the removeOnCompleted/Failed options for this.