ideas
ideas copied to clipboard
Allow pausing batches
What
An ability to store a batch in the database and control when it should be executed.
It can be implemented by pause
and continue
methods.
Why
I, for example, want to dispatch 5,000 jobs in a batch. Every job contains a User
model with many relationships. Fetching these models consumes much memory, so I must split the fetching into a few chunks, and push every chunk to the existing batch.
But the problem is that the batch is already running. There's a race condition problem here: the workers might finish the pending jobs before the next chunk is fetched and pushed to the batch - and this will result in completing the batch before it really gets finished.
If there was an option to pause the existing batch, and start it just after the whole chunks got stored - it will fix the race condition problem.
Example - Current state
$batch = Bus::batch([])->allowFailures()->dispatch();
User::query()
->addManyRelations()
->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
$jobs = $usersChunk->map(static function (User $user) {
return new SendNotificationToUserJob($user);
});
// The `$batch` might be finished before the second chunk is fetched.
$batch->add($jobs);
});
Example - After implementing pausing batches
$batch = Bus::batch([])->allowFailures()->dispatchPaused(); // or `->dispatch()->pause()`
User::query()
->addManyRelations()
->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
$jobs = $usersChunk->map(static function (User $user) {
return new SendNotificationToUserJob($user);
});
$batch->add($jobs);
});
$batch->continue();
- I'm not sure "pause" is the best term for this feature. But the ability to decide when a batch should be executed is what I meant.