Global Concurrency Setting for Bull-MQ Workers

[faisal-merchlink]

I hope this message finds you well. Currently, we are facing a challenge in our system where the need for sequential execution of certain processes conflicts with Bull-Mq's default behaviour, particularly when the server is running on multiple instances. This situation has prompted us to seek an enhancement in Bull-Mq to better support our use case.

// First instance of the server
const queueName = 'SampleQueue';
const queue = new Queue(queueName, {
  connection: {
    host: '127.0.0.1',
    port: 6379
  },
});

// Adding jobs to the queue
queue?.add('JobA', { 'objectId': 1 });
queue?.add('JobB', { 'objectId': 2 });
queue?.add('JobC', { 'objectId': 3 });

// First instance of the server worker
new Worker(
  queueName,
  async (job) => {
    console.log('Job started in the first instance');
    return true;
  },
  {
    connection: {
      host: '127.0.0.1',
      port: 6379
    },
  }
);

// Second instance of the server worker
new Worker(
  queueName,
  async (job) => {
    console.log('Job started in the second instance');
    return true;
  },
  {
    connection: {
      host: '127.0.0.1',
      port: 6379
    },
  }
);

The first instance of the server is adding jobs to the queue, and both the first and second instances of the server workers are executing these jobs sequentially. However, I want to have a global concurrency setting for the workers.

[manast]

The concurrency setting can only be set on each worker, so if you have two workers then it will add the concurrency factor of every worker, in the example above the total concurrency would be 2. Can you explain a bit more what is the context for having a global concurrency setting?

[faisal-merchlink]

The concurrency setting can only be set on each worker, so if you have two workers then it will add the concurrency factor of every worker, in the example above the total concurrency would be 2. Can you explain a bit more what is the context for having a global concurrency setting?

Thank you for your response. In our system, certain critical processes require sequential execution, and the current behavior of Bull-MQ poses a challenge in maintaining this sequence when the server is running on multiple instances. While we appreciate the concurrency settings on individual workers, having a global concurrency setting would simplify the management of sequential processes across all instances of the server. This is particularly crucial for processes that depend on a specific order of execution. Having a global concurrency setting would allow us to ensure that, regardless of the number of instances, these processes are executed sequentially, meeting our system requirements more effectively. We believe this enhancement would greatly enhance the flexibility and usability of Bull-MQ in scenarios like ours. Thank you for your consideration.

[manast]

@faisal-merchlink It is possible to achieve what you are asking with BullMQ Pro. You need to use the groups feature and set the maximum concurrency to 1, that will effectively imply that maximum one job for that group would be executed independently on the number of workers: https://docs.bullmq.io/bullmq-pro/groups/concurrency There is one caveat though, if a job fails and you have retries with some delay, while the failed job is delayed, the next job would start processing. We are thinking on a new setting to also take account for that.

[manast]

This PR should address this issue: https://github.com/taskforcesh/bullmq/pull/2465

[roggervalf]

Hi @faisal-merchlink for global concurrency, It's already addressed in this pr https://github.com/taskforcesh/bullmq/pull/2496