taskforcesh
When bullmq was deployed to k8s, Concurrency was out of whack!!!
Hi master,
When I deploy to k8s, create two pods and set Concurrency to 2. Normally, each pod can process two jobs concurrently, but in practice each pod can process four jobs concurrently, and it was found that the set number of Concurrency times the number of pods, which affects Concurrency,
Could you give me some idea? please.
hi @fifa334,
to understand your case, do you have 2 workers in each pod with concurrency 2?
we have 2 ways to set concurrency, there is a local concurrency https://docs.bullmq.io/guide/workers/concurrency#local-concurrency-factor how many jobs are allowed to be processed in parallel for that instance and we also have a global concurrency option https://docs.bullmq.io/guide/queues/global-concurrency determines how many jobs are allowed to be processed in parallel across all your worker instances
hey @fifa334, For case 1 we might need to have a test case to see that behavior, as I can tell local concurrency should work, but if you see 2 jobs in active state, could you verify that there are no more workers than just 1 for queueA, you can use this method https://api.docs.bullmq.io/classes/v5.Queue.html#getWorkersCount? For case 2, it's working as intended, global concurrency value is how many jobs can be executed at the same time cross worker instances. If you set it as 1, only 1 job will be processed at any time, after it is finished, next job will be moved to active. What are your expectations in this case?
@fifa334 could you please give us a code snippet that reproduces your issue?
@fifa334 I think you have some misconceptions on how workers and concurrency work in BullMQ, so your questions are not really making a lot of sense :)
To summarise. 1 Worker instance (new Worker(...)) can process as many jobs concurrently as the concurrency you use for that worker. You can have as many worker instances as you want, every worker instance will process as many jobs at the same time as its concurrency setting. This is covered here: https://docs.bullmq.io/guide/workers/concurrency
There is a special option though, called "global concurrency", where you can specify the max amount of jobs processed concurrently by ALL your workers. So if you set this value to 2 for example, then independently on how many workers and what concurrency setting you use, the workers will only process 2 jobs at the same time. https://docs.bullmq.io/guide/queues/global-concurrency