taskforcesh
[Bug]: Method upsertJobScheduler does not trigger the new scheduling immediately
Version
v5.63.0
Platform
NodeJS
What happened?
I'm using version v7.40.0 of BullMQ PRO, that is using version v5.63.0 of bullmq.
When I update an existing scheduled job using queue.upsertJobScheduler and change the "every" interval (for example, from "every 10 seconds" to "every 2 seconds"), the new schedule is applied, but the job only starts running with the new interval after the previous schedule time has elapsed. In other words, the job does not reschedule immediately after the update, but only after the original schedule interval has finished.
How can I force the new scheduling to be taken into account immediately? I thought that this issue could be related to https://github.com/taskforcesh/bullmq/issues/3500 but it seems fixed from v5.62.2...
What am I doing wrong? What is the right way to achieve what I need?
How to reproduce.
No response
Relevant log output
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
After some time anylizing this issue, one question to be answered is what would be the actual correct behaviour. If you upsert a job scheduler with a new "every" setting, when should the next job be scheduled? should it be processed 2 seconds from the moment the new job scheduler is executed, or right at the moment it is executed? Currently when you add a job scheduler for the first time (with every setting), the first iteration will be as soon as possible and then it will repeat according to the every setting. But if you are updating an existing job scheduler, what would be the desired behaviour?
In my opinion, when we're updating a job already scheduled with upsertJobScheduler:
- by default you could leave the behavior as is: the job scheduling is actually changed, but before executing the job again it will wait the "old" scheduling once (after 2 seconds) and then will start the new scheduling (each 10 seconds)
- it could be useful to add a couple of flags/options to the
upsertJobSchedulermethod- one to force the immediate execution of the job and then apply immediately the new scheduling: in this case a job originally scheduled each 2 seconds, when changing interval to 10 seconds the job would be executed once NOW and then each 10 seconds from NOW
- one to just apply immediately the new scheduling, starting from NOW: in this case a job originally scheduled each 2 seconds, when changing interval to 10 seconds the job would NOT be executed NOW but correctly after each 10 seconds from NOW
Finally, what you wrote for the addition: when you add a job scheduler for the first time (with every setting), the first iteration will be as soon as possible and then it will repeat according to the every setting the problem is that I would expect it to act like this only with option immediately set to true, but even setting it to false, the job is executed immediately after the addition. In my opinion, setting option immediately to false, it should NOT execute the job immediately, but wait the next scheduling interval starting from NOW if we use the "every" option, exactly in the same way as it happens when using the "pattern" option.
The "immediately" flag was removed for the every setting after a lot of deliberation. The rationale is that the most common desired behaviour is that it starts repeating directly from the moment the job scheduler is added, and if you want any other behaviour you can just use "startDate" to choose from when it should start in the future. I think this is the easiest to understand (and also implement) behaviour.
I think the most logical behaviour for the case that you are reporting in this issue would be to schedule the next job now, but taking into consideration one edge case, namely that the last job iteration also happened to be processed now. In this case the job will be scheduled to the next time slot. This should work for most cases I think.
The "immediately" flag was removed for the every setting after a lot of deliberation. The rationale is that the most common desired behaviour is that it starts repeating directly from the moment the job scheduler is added, and if you want any other behaviour you can just use "startDate" to choose from when it should start in the future. I think this is the easiest to understand (and also implement) behaviour.
Ok for this part, it seems reasonable.
I think the most logical behaviour for the case that you are reporting in this issue would be to schedule the next job now, but taking into consideration one edge case, namely that the last job iteration also happened to be processed now. In this case the job will be scheduled to the next time slot. This should work for most cases I think.
I'm not sure I understand this part very well. Could you kindly explain it with an example?
I'll try to explain what I expect, so that we can share our common understanding on this case: let's consider a job scheduled each 2 minutes that I want to be re-scheduled after each 10 minutes. Let's say that now it's 14:48:41 and that the original scheduling (after each 2 minutes) should run at 14:50:00. If I change to 10 minutes, what I expect is for it to be launched now, let's say 14:48:42 and then the next iteration should be 14:58:42, then 15:08:42, ...
Is it this what you're saying? What I don't understand is this phrase taking into consideration one edge case, namely that the last job iteration also happened to be processed now
I'm not sure I understand this part very well. Could you kindly explain it with an example? I'll try to explain what I expect, so that we can share our common understanding on this case: let's consider a job scheduled each 2 minutes that I want to be re-scheduled after each 10 minutes. Let's say that now it's 14:48:41 and that the original scheduling (after each 2 minutes) should run at 14:50:00. If I change to 10 minutes, what I expect is for it to be launched now, let's say 14:48:42 and then the next iteration should be 14:58:42, then 15:08:42, ... Is it this what you're saying? What I don't understand is this phrase
taking into consideration one edge case, namely that the last job iteration also happened to be processed now
yes, thats what I am saying, but imagine that when you change to 10 minutes, instead of being 14:48:42 it happens to be at exactly 14:50:00 and that the previous job managed to already started to be processed at that exact time. In this case the next job would be at 15:00:00.
imagine that when you change to 10 minutes, instead of being 14:48:42 it happens to be at exactly 14:50:00 and that the previous job managed to already started to be processed at that exact time. In this case the next job would be at 15:00:00.
Yes, but in this case it would still be correct, right? It is again executed "now" and then "after 10 minutes".
depends, if you changed also the data or the options of the job scheduler, the now will execute the old data and options instead of the new one. But I do not think that in general this can be made more robust than this to be honest.
Yes, I understand, but the NEW iteration (not the current one) will use the right data and options, right? In that case, let's go on with that change then.
One more thing, just to be sure: what will happen if the scheduling is made with a pattern (e.g. CRON syntax scheduling) instead of an every option? For example, let's say that a job is scheduled on Tuesday at 15:00 and now it's Tuesday at 09:00. If I change the scheduling to Tuesday at 16:00 I imagine that after the change, the job will NOT be executed immediately but will wait Tuesday at 16:00 (while the old scheduling of Tuesday at 15:00 is skipped). Can you confirm that too?
pattern is scheduled according to the pattern unless "immediately" is set to true.
Ok @manast. Do you think you could deliver this fix on the next days and for the next PRO version?