simplekiq icon indicating copy to clipboard operation
simplekiq copied to clipboard

Limited parallelization for in_parallel and/or BatchedJob

Open jcwilk opened this issue 2 years ago • 0 comments

https://github.com/doximity/campaigns/blob/f3b5a7d457a0fc4b5e310c851ac9b55570ce18b2/app/jobs/audiences/sleepy_mass_cache_orchestration_job.rb

^ this job takes an approach to limiting the parallelization by recursively queuing itself after the partial set has completed, we could probably use a similar approach transparently in our batching behaviors, perhaps like in_parallel(10) or a sidekiq option in BatchedJob or something like that.

This is useful for avoiding flooding the queue so that new jobs can still interleave with the orchestrated batch.

jcwilk avatar May 02 '22 22:05 jcwilk