ruby-shoryuken
Batching Support in Rails Active Job
I could not find any way to implement batch fetching from SQS in active jobs. Am i missing something? Is batching not supported in active jobs and if yes, was there a particular reason for it?
Hey, new maintainer here. I am not yet fully onboarded but this seems like a deliberate decision. ActiveJob was designed with a single-job processing model in mind. Its architecture assumes jobs are processed one at a time, which aligns with many of its adapter backends. While Shoryuken itself supports batch processing ActiveJob adapter doesn't expose this functionality..
If batch processing is important for your use case (which makes sense for performance with SQS), you have a few options:
- Use Shoryuken directly instead of through ActiveJob for the specific jobs that need batching
- Create a custom Shoryuken worker that handles batches and then delegates to your ActiveJob jobs
- Consider contributing to the Shoryuken ActiveJob adapter to add batch support (the community might appreciate this)
can you elaborate more on point number 2, what exactly does it mean and how to do it?
There's the other side to batch jobs. Batch enqueueing.
This has been brought up a couple of times
- https://github.com/ruby-shoryuken/shoryuken/issues/710
- https://github.com/ruby-shoryuken/shoryuken/issues/761
But I see those issues timed out.
They timed out with previous maintainer. Once Im back from RailsConf (next week) I will be doing a major rework around several things including this
They timed out with previous maintainer. Once Im back from RailsConf (next week) I will be doing a major rework around several things including this
@mensfeld curious if you have an update. Next week I'm starting to explore ways to batch messages to SQS . We have a need to make enqueueing messages quicker. ~250k messages.
For context, we are using shoryuken with active job, utilizing FIFO queues.
Thanks!
@davidrichey the only update i have is that I did not forget about it but I cannot give you ETAs because I started work on removing concurrent ruby to switch to a manual thread management and few other shoryuken related things.
If this is important to your business
We have a need to make enqueueing messages quicker. ~250k messages.
you can consider supporting my work via a monthly Stripe payment (like I do with Karafka) and then I can prioritize such work. Otherwise you are free to submit a PR and I will gladly review it.