[Bug]: BullMQ Throws 'Connection is Closed' Error with ioredis Cluster on Disconnect

[nikolozz]

Version

v5.1.5

Platform

NodeJS

What happened?

We encountered an issue with BullMQ when using the ioredis module (version 5.3.2) in cluster mode. Specifically, when attempting to close the Queue and after that Redis connection BullMQ generates the error message "Error: Connection is closed." This error arises during the execution of waitUntilReady method.

When using the Ioredis cluster the status names of the Redis client are different, for example, if I'm using a single connection to the Redis, after the connection status will be "ready" which will execute the following condition:

  if (client.status === 'ready') {
      return;
  }

However, the status for a Cluster client is different and is typically 'connect' instead of 'ready'. this status will not be handled in the waitUntilReady method, which will cause the error.

How to reproduce.

Create the queue:

import { Cluster } from "ioredis";

const cluster = new Cluster(
        [
          { host: '127.0.0.1', port: 6379 },
          { host: '127.0.0.1', port: 6380 },
          { host: '127.0.0.1', port: 6381 },
          // ...
        ],
        { enableOfflineQueue: false },
      );

const queue = new Queue("queue", {
      connection: cluster,
      prefix: '{prefix}',
    })

Close the queue and redis gracefully:

process.on('SIGTERM', async () => {
  await queue.close();
  await cluster.disconnect();
})

Notice the error (provided in log output)the

Relevant log output

Error: Connection is closed.
    at EventEmitter.handleEnd (/Users/macos/test-project/node_modules/bullmq/src/classes/redis-connection.ts:181:4)
    at EventEmitter.emit (node:events:525:35)
    at /Users/macos/test-project/node_modules/ioredis/built/cluster/index.js:578:18
    at processTicksAndRejections (node:internal/process/task_queues:77:11)

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[kibertoad]

@manast ~This is due to the fact that Bullmq closes the Redis connection when closing the worker, which seems like a rather questionable decision - more often than not Redis connection is reused across multiple workers, and sometimes even outside the BullMQ.~

~Would it be possible to either add a parameter to worker.close() method that would make it possible to avoid closing the underlying Redis connection, or a separate method for closing only the worker but not the connection?~

~We can send a PR if you are OK with the change and have a preferred way of doing it.~

See https://github.com/taskforcesh/bullmq-proxy/pull/7#discussion_r1501848558

[manast]

@kibertoad the problem is that what you are saying is not really true... only connections that are not shared are closed.

[semoal]

We're seeing this error too, running Redis in normal mode (Not cluster) and executing await queue.close() before returning our api request value. We connect to Redis with the connection object:

export const DEFAULT_CONN: QueueOptions = {
  connection: {
    port: process.env.REDIS_PORT ? +process.env.REDIS_PORT : 6379,
    host: process.env.REDIS_HOST,
    username: "default",
    password: process.env.REDIS_PASSWORD,
    family: process.env.REDIS_HOST?.includes("internal") ? 6 : 4,
    db: 0,
  },
  defaultJobOptions,
};

/Users/sergiomoreno/Projects/sherpas/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/redis-connection.js:100
                    reject(lastError || new Error(utils_1.CONNECTION_CLOSED_ERROR_MSG));
                                        ^

Error: Connection is closed.
    at EventEmitter.handleEnd (/Users/sergiomoreno/Projects/x/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/redis-connection.js:100:41)
    at EventEmitter.emit (node:events:514:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
Emitted 'error' event on RedisConnection instance at:
    at /Users/sergiomoreno/Projects/x/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/redis-connection.js:61:45
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Should I do this for avoiding the unhandled crash, we've been seen this error since upgrading to 5.4 from 4.7, so not sure if it's correct or not.

  const queue = new Queue(Queues.MAILING, DEFAULT_CONN);
  const redis = await queue.client;
  redis.on("close", () => {
    console.log("Connection closed");
  });
....

await queue.close()

[manast]

@semoal can you provide reproducible code so that we can look into it?

[semoal]

Yes, will prepare a repro and ping you back