simplekiq
simplekiq copied to clipboard
Limited parallelization for in_parallel and/or BatchedJob
https://github.com/doximity/campaigns/blob/f3b5a7d457a0fc4b5e310c851ac9b55570ce18b2/app/jobs/audiences/sleepy_mass_cache_orchestration_job.rb
^ this job takes an approach to limiting the parallelization by recursively queuing itself after the partial set has completed, we could probably use a similar approach transparently in our batching behaviors, perhaps like in_parallel(10)
or a sidekiq option in BatchedJob or something like that.
This is useful for avoiding flooding the queue so that new jobs can still interleave with the orchestrated batch.