Planned batch support?

[jpcamara]

First off, congrats on the initial release! I'm really excited to start giving this a try - it includes a very robust feature set out of the gate which is amazing!

The docs mention features coming very soon:

Proper support for perform_all_later, improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon.

Have you had any internal discussions or thoughts on batch support? By batch support I mean like GoodJob batches (https://github.com/bensheldon/good_job?tab=readme-ov-file#batches) and Sidekiq Batches (https://github.com/sidekiq/sidekiq/wiki/Batches). They're a huge benefit for job coordination.

I'd be very interested in contributing something like this, if there were no plans for it!

[rosa]

Thanks for your kind words, @jpcamara! I really appreciate it 😊

We have batch support in our list of possible features to add, but it's not in the immediate plans because it's a bit at odds with the simplicity we're aiming for with Solid Queue. If you'd like to try implementing it, you're, of course, welcome to do so! 😊 I'm not quite sure yet how it could look like, in a way that maintains the overall simplicity of the gem, so I'm all open to ideas!

Thanks again!

[mbajur]

Would be amazing to have this feature implemented in solid queue. Keeping my fingers crossed for it 🤞

[McRip]

This is one of the features which makes us rely on sidekiq currently. I would also like to see this being implemented in solidqueue.

[matteeyah]

@jpcamara You could take a look at https://github.com/cdale77/active_job_status. It's fairly simple, if it doesn't work with SolidQueue, it shouldn't be too hard to update it to work with SolidQueue as well.

[matteeyah]

I've been working on https://github.com/inkstak/activejob-status/pull/32 and I just noticed https://github.com/basecamp/solid_queue/commit/5ad872727251c32c282a7eb44df1e24068a3fdaa landed in main.

It could be a relatively simple way to approach batches - a batch could just be a collection of a job statuses - with different combinations of statuses adding up to different final batch statuses.

Here's my idea - The batch status can be queued, failed, completed or working.

1. The batch is considered queued if all of the jobs are queued
2. The batch is considered failed if one of the jobs is failed
3. The batch is considered completed if all of the jobs are completed
4. The batch is considered working in all other circumstances

---

On one hand, I dislike the solution we're currently using (with https://github.com/inkstak/activejob-status), because it relies on the cache for functionality - without the job status being stored in the cache, the application would simply break.

---

On the other hand, SolidQueue::Job#status is implementation specific, and like @rosa said, it's

This is prone to race conditions and stale reads, and it's just intended
for use in Mission Control or other reporting tools, not for regular
operation.

---

Even though using the collection of job statuses is a simple and effective approach, I'm still not sure what would be the ideal way forward here.

[jpcamara]

Hi @rosa! I've opened a draft PR to introduce how I think batches could be implemented in SolidQueue: https://github.com/basecamp/solid_queue/pull/142

Looking forward to feedback on whether it's the right direction or not, when you get some time!