Redis Cluster, Can't add the job to the failed queue

[wdh-1025]

In the environment of Redis cluster,Can't add the job to the failed queue,Job will hold the lock for a long time until the Job times out!

Description

queue.process('testjob', async (job,done) => {
     done(new Error('error transcoding'))
     //OR
     throw new Error('error transcoding')   
     //Can't add the job to the failed queue
})

Bull version

3.22.0

[manast]

You must not use "done" and async at the same time in your process functions. Furthermore if you still have issues provide a complete reproducible case so we can look into it.

[wdh-1025]

Thanks for the reply, but this still doesn't seem to solve my confusion！

const bull = require('bull');

const queue = new bull('cluster', {
    prefix: '{test}',
    redis: {
        port: 6401,
        host: '127.0.0.1',
        password: 'He2jY9K3LNplLmbFS50i',
    },
    defaultJobOptions: {
        attempts: 3,     
        removeOnComplete: true,
        backoff: false
    },
    limiter: {   
        max: 1000,
        duration: 5000,
        bounceBack: true
    },
    settings: {
        maxStalledCount: 1,
        guardInterval: 3000,
        retryProcessDelay: 500
    }
})

queue.add('test', { event: 'access_token', target: 'public' } )

queue.process('test', (job,done) => {
    //Can't add the job to the failed queue
    done(new Error('test error'))
})

[manast]

A couple of things, did this same example work in non cluster redis? and secondly, how do you know the job did not end in the failed status?

[wdh-1025]

It can work normally on non-cluster! The job holds the lock for a long time after the error is thrown, and it is not added to the failed queue until a long time later (job stalled more than allowable limit)

[manast]

ok, then you need to define the prefix for the queue like this: "{myPrefix}", default is "bull" which is nt compatible with cluster: https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#redis-cluster

[wdh-1025]

As in the above case, I have defined the prefix, but still cannot achieve the expected effect，So I don't know if this is a framework compatibility issue. Is it possible to fix it in the future?

[manast]

As in the above case, I have defined the prefix, but still cannot achieve the expected effect，So I don't know if this is a framework compatibility issue. Is it possible to fix it in the future?

oh yes sorry, I am blind this morning. What is your redis cluster setup?

[wdh-1025]

I am not quite sure what you mean, Because my Redis is a purchased cloud service. Redis Configuration(Redis 5.0 standard edition, dual copy, 2G cluster edition (2 shards))!

[manast]

I need some setup in order to test it myself... is this in RedisLabs or where?

[wdh-1025]

The Redis service I purchased from ALiYun

[manast]

Ok, I will try to reproduce this in a local setup. However, if you do not need the cluster setup for other things than Bull then you could also switch to non cluster mode, since there are no advantages in using cluster with Bull.

[wdh-1025]

Yes, but due to special reasons, I cannot switch to a non-clustered state. I look forward to making progress as soon as possible. Thanks again!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[davidnazw]

We're experiencing the same issue, I was wondering if this is planned to be fixed.

[manast]

@davidnazw first I need to be able to easily reproduce it, I haven't so far...

[zangguojun]

Yes, but due to special reasons, I cannot switch to a non-clustered state. I look forward to making progress as soon as possible. Thanks again!

老哥 我也遇到同样的问题了 请问 你解决了吗？

[manast]

As a side note, you may consider upgrading to BullMQ -> https://bullmq.io as there are no know issues with Redis Cluster.