A way to remove the "on the hour" behavior of the Repeatable Job

[gitChristian]

I'm wondering if there is a way to remove the "on the hour" behavior of the Repeatable Job specified here.

Adding a job with the repeat option set will actually do two things immediately: create a Repeatable Job configuration, and schedule a regular delayed job for the job's first run. This first run will be scheduled "on the hour", that is if you create a job that repeats every 15 minutes at 4:07, the job will first run at 4:15, then 4:30, and so on.

Use case: I have a queue with jobs that should run every 6 hours. Throughout that time window, many jobs will be added to the queue. Because of this "on the hour" behavior, all the jobs will try to be run at the same time. Is there any way to have them staggered? Thanks!

[gitChristian]

One solution im thinking of is this:

    await refreshQueue.add( id, { data },
      { repeat: { every: minutes * 60 * (800 + <random number 1-400>, } } //800 represents milliseconds
    );

This will create some staggering, but is there something better?

[gitChristian]

This is the wrong repo for BullMQ issues, but I can give you an answer, you can use cron expressions to do this so that you generate the expression in such a way that it is based on the moment the job was added, then you can accomplish staggering.

Thank you @manast . However, I tried this approach with startDate and cron expressions. Even with this approach, I was getting "on the hour" behavior. All the jobs were clumping together. My approach with random numbers is the only thing I could get to work.

[gitChristian]

    const ttlHours = Math.floor(ttlMinutes / 60);
    const ttlMilliseconds = ttlMinutes * 60 * 1000;
    const cron = ttlHours < 24 ? `* * */${ttlHours} * * *` : `* * * */1 * *`;
    await refreshQueue.add(id, { data },
      { repeat: { cron: cron, startDate: Date.now() + ttlMilliseconds }, }
    );

[umutm]

I have recently started using BullMQ and wondering if any other solutions to remove the "on the hour" behavior?

Random numbers solution causes each job to be processed every "unknown" seconds as the random number differs each time.

Any way to make a job to be ran with intervals like 18:02, 18:07, 18:12?

P.S> This behavior makes handling bulk jobs pretty difficult. Other than that, awesome MQ.

Thanks a lot.

[manast]

I got this suggestion by ChatGTP:

Yes, you can achieve the behavior you're describing using cron expressions in BullMQ, or any other job scheduling system that supports cron-like scheduling. Cron expressions are powerful for scheduling tasks at specific intervals or specific times, and they can be used to schedule jobs exactly as you described, like at 18:02, 18:07, 18:12, and so on.

For the scenario you've mentioned, wanting a job to run at intervals such as every 5 minutes past the hour starting from 18:02, you can use a cron expression that specifies this. Unfortunately, standard cron expressions do not support specifying jobs at irregular minute intervals like every 5 minutes starting at a minute other than 0 or 5. They are designed to run at fixed intervals (e.g., every 5 minutes starting at the hour or half-hour).

However, you can achieve your goal by scheduling multiple jobs with specific cron patterns for each or by implementing a custom scheduling logic within your application. Here's a workaround using multiple cron jobs:

1. Schedule Multiple Jobs: You would schedule a job for each specific time you want it to run. For example, to run a job at 18:02, 18:07, 18:12, etc., you would create separate schedules for each.

For a job to run at 2 minutes past every hour, you can use:

2 * * * *

For a job to run at 7 minutes past every hour, you can use:

7 * * * *

And so on.

2. Custom Scheduler: Implement a custom scheduler in your application that calculates the next execution time based on your specific logic and reschedules the job accordingly after each run.

If you are strictly looking for a single cron expression to handle this, due to the constraints of cron syntax, you won't be able to specify an irregular pattern like "every 5 minutes starting at 2 minutes past the hour" directly. You would either need to set up multiple schedules or implement custom logic to dynamically adjust the next execution time of your job.

Given these constraints, it often becomes a trade-off between managing multiple cron jobs for each specific time or implementing more complex logic within your application to handle custom scheduling needs. The approach you choose will depend on your application's complexity and your preferences for managing scheduled jobs.

[umutm]

@manast Thanks very much for this update and info.

I was not aware that this was cron's default behavior and it all makes much more sense now.

I'll probably be re-scheduling jobs via setTimeout by adding x, y, z.. milliseconds to them to distribute the load.

Thanks very much again.

[manast]

With the new API I hope this issue can be easily accomplished, either using cron expressions or a custom repeat strategy: https://docs.bullmq.io/guide/job-schedulers