Bulk / Batch Processor worker - process many jobs at once

[mariuszbeltowski]

Is your feature request related to a problem? Please describe. Example case: Application receives 3k events from API per second. We want to store them in the database, but processing every event as single insert would generate 3k/s inserts and heavily loads the db. The solution is to use bulk insert, but to use that we need a way to read jobs in a bulk way inside the processor. This is a common practice with other queue systems.

Describe the solution you'd like Allow the processor to receive many jobs in once instead of only one.

Describe alternatives you've considered

Additional context

[roggervalf]

hi @mariuszbeltowski, we support batch processing in pro version, pls take a look https://docs.bullmq.io/bullmq-pro/batches

[mariuszbeltowski]

hi @mariuszbeltowski, we support batch processing in pro version, pls take a look https://docs.bullmq.io/bullmq-pro/batches

Thank you @roggervalf . How it will work if we set 10 items per batch, but on the queue we will get just 8? Is there any timeout for fulfilling the worker for items.length < batch? Can we control that timeout? I couldn't find that in the docs.

[manast]

@mariuszbeltowski it will take up to 10, if there are less than 10 then it will take all the ones that are left.