Debouncing/Unique Jobs

[AaronBetterOfficeApps]

I want to build some middleware to handle debouncing jobs on the queue. The resulting behaviour would be that a job for the same worker with the same args would only be ran once every configured time interval and it would be the last job to enter the queue.

It could work something like the following with a 1 second debounce

• Job enters the queue
• A time stamp is stored with metadata of worker and the args
• Job waits for 1 second
• Query the time stamp for the worker and the args
• If no other job has entered the queue and bumped the times stamp the job runs
• If the timestamp is greater than 1 second ago then job aborts, allowing for the job that reset the timestamp to be the job that is ran

I have a few idea's of how this could be implemented. One way would be to use redis and have the worker and the args as the key. Then, set an expiry on the key so that if the key is not there after the job sleeps, it will run. However, a new job would reset the expiry when it enters the queue. Or, this could be handled with some kind of internal metadata store using ETS.

What would be the preferred method of storing the metadata to make this kind of middleware possible?

[akira]

@AaronBetterOfficeApps I'm assuming you would want unique jobs across the whole cluster? If you're running jobs across multiple instances of your Elixir app, storing them in ETS would not work, you would have to store them in Redis.

[nicooga]

I have this same problem.

The ruby solution using sidekiq was:

• if JID is present on record cancel that job
• enqueue the job to run X time after and obtain JID
• store the JID on the DB

But I can't find a method of cancelling a job on exq, is there any?

[nicooga]

I see there is a dequeue function in Exq.Redis.JobQueue. There is no interface in Exq.Enqueuer.EnqueueApi to use it though. Is there a particular reason to hide this feature?

[tanweerdev]

I also want to acheive unique jobs feature (which is also available as gem https://github.com/mhenrixon/sidekiq-unique-jobs). @nicooga if you have found a workaround please do share

[nicooga]

@tanweerdev I need a job to run once X time after the last occurrence of an repeating event. I'm not sure debouncing is the right term to describe it.

[atomkirk]

Would love to see this too.

What I did was create a pool of dynamic queues, each with concurrency 1, and I mod on the id integer argument to decide which queue to add it to. The same id will always be added to the same queue.

[hez]

We are also interested in this.

[OpakAlex]

any news?