Set the failed/completed jobs automatically removed after a period time

[Tin-Nguyen]

Hi all,

Do we have an option in Bull allows to remove the failed/completed jobs in queue after a period time (1 day/week/month)? I think it's necessary because it can reduce the number items in Redis storage.

Thanks in advanced. Tin

[mscharley]

For what it's worth, we just run queue.clean(3600 * 1000, "completed") once a minute and this means we only have an hours worth of completed tasks at any point in time.

[Tin-Nguyen]

@mscharley thanks. But I'm wondering why we don't have a function to let us be able to set a permanent process for the queue to remove the jobs after a period time? Could we consider to have an improvement on this?

[mscharley]

Sure, and I 👍 'd this issue too. Just letting people know of workarounds in the meantime.

[aleccool213]

We store our job details elsewhere so this is just duplicated data being stored. This is a much wanted feature from me as well :)

[sriharshakappala]

@mscharley I wanted to clear jobs from failed state and I tried queue.clean(3600 * 1000, "failed") and it didn't work. What's the right way?

[axos88]

Without this feature the amount of data in this increases to infinity, unless one of the workarounds is implemented of course.

It would be really great if it was possible to retain statistics (number of jobs processed), without having to keep the actual job details around.

[addisonElliott]

There is a removeOnCompleted & removeOnFailed option for each job that will allow removing once the job is complete or failed. This should work in doing what needs to be done.

As for retaining statistics for jobs without keeping the jobs around, I think that's relatively easy to do in userland by tracking the 'completed' and 'failed' events on the queue. Just keep increment some value in redis.

[axos88]

The removeOnCompleted and removeOnFailed would remove the information immediately. I'm looking for a solution that would keep that around for an amount of time, so that they can be reviewed for say 7 days, and then scrapped so they don't clog up the redis storage.

The problem with implementing the statistics in userland is that there is already statistics provided by bull, and that info would become incorrect when deleting (old) jobs.

[manast]

@axos88 it does not remove it immediately, please check the documentation: https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd

[axos88]

@manast you can keep around "X" number of items, but cannot set a time limit for them. If I want to keep them around for 1 week, and use a workaround of setting removeOnCompleted to say 5, and there are only one job per week of that type i will end up keeping them around for 5 weeks instead. If there are one job per hour, i will end up keeping them around for only 5 hours. This workaround works if you have existing knowledge of how often the job will run and can determine an accurate guess of how many occurances to keep around, such as a scheduled cronjob, but does not work for something that is triggered by your users' behaviour, such as sending a message, or placing and order.

But you're right in that what I am gunning here is actualy to extend that syntax to allow boolean | number | Date.

[manast]

yes, however Date is also a sword with two edges, if you retain based on date you may end filling up all redis memory.

[axos88]

@manast, Well yes, of course, but that's a decision that needs to be weighed by the user/programmer and the approariate action needs to be taken. In my case I know that I can hold the data of a week in memory, but would run out if left unchecked for months or weeks.

And of course the two can be combined saying hold the data of a week, but up to maximum 100 jobs, or the other way around keep up to 100 jobs in redis, but delete them after a week, even if there are less than 100 to clean up unuseful data.

[wodCZ]

For the reference, this seems to be implemented, if I understand this issue correctly. https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#keepjobs-options, https://github.com/OptimalBits/bull/blob/4b8a386acedcec6da137f9bb810170c3a8eb11d2/lib/commands/moveToFinished-9.lua#L119-L132

{
    removeOnComplete: { age: 86400 * 5 }, // remove completed jobs after 5 days
}