ideas icon indicating copy to clipboard operation
ideas copied to clipboard

Allow pausing batches

Open MatanYadaev opened this issue 3 years ago • 0 comments

What

An ability to store a batch in the database and control when it should be executed. It can be implemented by pause and continue methods.

Why

I, for example, want to dispatch 5,000 jobs in a batch. Every job contains a User model with many relationships. Fetching these models consumes much memory, so I must split the fetching into a few chunks, and push every chunk to the existing batch. But the problem is that the batch is already running. There's a race condition problem here: the workers might finish the pending jobs before the next chunk is fetched and pushed to the batch - and this will result in completing the batch before it really gets finished. If there was an option to pause the existing batch, and start it just after the whole chunks got stored - it will fix the race condition problem.

Example - Current state

$batch = Bus::batch([])->allowFailures()->dispatch();

User::query()
    ->addManyRelations()
    ->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
        $jobs = $usersChunk->map(static function (User $user) {
            return new SendNotificationToUserJob($user);
        });

        // The `$batch` might be finished before the second chunk is fetched.
        $batch->add($jobs);
    });

Example - After implementing pausing batches

$batch = Bus::batch([])->allowFailures()->dispatchPaused(); // or `->dispatch()->pause()`

User::query()
    ->addManyRelations()
    ->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
        $jobs = $usersChunk->map(static function (User $user) {
            return new SendNotificationToUserJob($user);
        });

        $batch->add($jobs);
    });

$batch->continue();
  • I'm not sure "pause" is the best term for this feature. But the ability to decide when a batch should be executed is what I meant.

MatanYadaev avatar Jul 12 '21 12:07 MatanYadaev