rq
rq copied to clipboard
rq
Detect duplicate jobs
Besides manually maintaining a list outside of RQ someplace, is there a good built in way to ensure that a job isn't queued while an identical job is still waiting to be processed?
There's a branch containing a job Registry class that maintains a list of enqueued jobs (and their functions). Once this is merged in, we'll be able to do:
if func in queue:
# Do something
Will that check with parameters? For my purposes, I'd want to check the combination of the function and all arguments.
Not at the moment. You'd have to extend it by getting all jobs having that function (a cheap operation) and checking for their arguments (may be expensive depending on the number of jobs you have).
@selwin As said before, it could be nice to have that as well:
if (func, (3,), {}) in queue:
# Do something
Under the hood, you could do the loop you're suggesting.
We could use job.get_call_string() instead of just job.func_name in the code that @selwin suggested, to achieve that. Isn't it?
what is the status of this deduplication? Is it in the current version of RQ?
@VidJa this feature is not yet implemented. This is something I look forward to tackling when I have the time. The first step would be to build a mapping of the types of enqueued jobs - this will be very useful for monitoring
