taskforcesh
Allow deduplicated jobs to queue when job is already running
Is your feature request related to a problem? Please describe.
I want to run a certain task in response to some user activity. However, it's an expensive task, and users could potentially do the triggering activity many times in rapid succession. I can achieve this pretty well with deduplication (https://docs.bullmq.io/guide/jobs/deduplication).
However, I want to be absolutely sure that I'll always run the task after the latest user action such that I never fail to process the latest state. Based on the docs, this doesn't seem to be the behavior:
While this job is not moved to completed or failed state, next jobs added with same deduplication id will be ignored and a deduplicated event will be triggered by our QueueEvent class.
I interpret this to mean that while my job is running, if I try to enqueue another job with the same deduplication ID, it will not be queued and will never run after the current one is done executing. This means that sometimes, I'll fail to process the latest data from the user.
Describe the solution you'd like
Back when debouncing (which ultimately became deduplication) was first proposed in https://github.com/OptimalBits/bull/issues/1034, what I'm describing was actually spelled out explicitly as a requirement for this new functionality: https://github.com/OptimalBits/bull/issues/1034#issuecomment-734181580. However, as far as I can tell, it was just never implemented (or if it was, it's not documented). For context, the functionality was implemented here: https://github.com/taskforcesh/bullmq/pull/2796
Describe alternatives you've considered
I'm not sure if there is one. Maybe it's already possible with the right set of flags and options? Or maybe things already work how they were originally specced to and the docs just don't reflect that? Any guidance would be appreciated.
Amazing timing. We ran into the need for this today and were surprised there was no support.
I'd encourage you to create other issues with your requests so we can keep the conversation narrowly focused on this one.
This would be something like instead of adding a standard job you would use a delayed job, the difference would be that instead of ignoring the next job that you add you would replace the previous delayed job and reset the delayed time. So if no new job is added before the delayed expires, the delayed job is effectively processed.
I'm not sure this entirely resolves this issue.
Correct me if I'm wrong, but it requires me to delay all jobs by a certain time period (TTL)?
What if I don't want to delay them?
I'd still want the job to be queued and executed as soon possible. However, if there's a incomplete job (that hasn't been started to be processed yet), I'd like to add it to the queue. (Even though that'd be a duplicate and could cause the same job to be processed twice in series/parallel.)
Is that currently possible in BullMQ?
@jtomaszewski not sure I understand your case completely, but it seems like what you want is just the basic case of deduplication based on jobIds? i.e. there can only be one job with a given id in the queue, if you add another job with the same id the second job will be ignored.
Hey @manast . I need:
- no arbitrary delays (run a job as soon as possible)
- never run the same job twice in parallel (instead, either skip processing it if there's already one in waiting state, OR, reprocess AFTER the currently active one finishes)
- ensure at least one job starts (and completes) AFTER a job schedule is requested.
Job IDs and current deduplication options don't help. https://github.com/taskforcesh/bullmq/issues/3427 would.
@jtomaszewski nor will work to enqueue a job if the previous one is in active status as any other worker will be able to start processing it before the previous one has completed. Which actual use case requires this functionality?