[Bug]: Different removeOnComplete options per job in the same queue

[marcdelalonde]

Version

5.1.1

Platform

NodeJS

What happened?

Hello,

I have a "webhooks" queue that is responsible of treating different webhooks.

Depending of the webhook type, I give a different removeOnComplete & removeOnFail config.

• webhook_type1 should be deleted after one week (on complete)
• webhook_type2 should be deleted after one day only (on complete)

const OneDay = {
  age: 3600 * 24,
};

const OneWeek = {
  age: 3600 * 24 * 7,
};

webhooksQueue.add("webhook_type1", data, { removeOnComplete: OneWeek, removeOnFail: false });
webhooksQueue.add("webhook_type2", data, { removeOnComplete: OneDay, removeOnFail: false });

But it seems that when the webhook_type2 is added, all the task in the queue are deleted if older than one day, event webhook_type1. webhook_type1 should be deleted after one week regarding the config I gave when adding it to the webhooks queue.

Is it a bug or normal behavior? Should I create different queue for each type of webhook? What is the most appropriate design with bullmq to manage it this way?

Thank you very much 🙏

How to reproduce.

No response

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[manast]

Yes, this works as "expected". As there is only one complete set, we just apply the removeOnComplete set for every job. So in this case the best would be to have 2 different queues, and I would also configure the removeOnComplete on the worker instead as it is more memory efficient.

[marcdelalonde]

Thank you for coming back to me @manast !

If I have for example 20 different jobs, do you advise to use 20 different queues, or dispatch them between some 4-5 queues depending of the removeOnComplete/removeOnFail config?

I have no idea what would be the best design for bullmq in term of performance and memory use.

Thank you

[manast]

I think it is better to keep the number of queues low, as it is easier to maintain. However, I wonder why you need different settings for the queues, could you share more details?

[marcdelalonde]

I receive millions of webhooks per month and some are very important to keep for a month, some others can be deleted just after completion. I have no choice to remove the second type of webhooks asap, as they can really fast sature the memory of my redis database. (as I received a big volume of them compared to the 1st type of webhooks)

I would like to keep it clean and efficient but im not very sure what the best design in my case.

[manast]

Sometimes it is useful to store information in a separate system, for instance you could store some data in a different Redis host or another database system with the result of the webhook processing.