Subscriber clients won't reconnect

[xairoo]

Subscriber clients won't reconnect after a timeout when redis server removes the client based on tcp-keepalive setting (default is 300 seconds).

This is only related to a remote redis connection. Locally it works. I ran into this because I put my system to sleep instead of shutting it down. But instead of sleep it could be a broken uplink, switch failure, power outage and so on.

The subscriber should automatically reconnect, but in this case he doesn't.

Reproduce

• Use a remote redis server
• Cut your uplink for at least 300 seconds (tcp-keepalive value) or put your system to sleep/hibernate
  • You can reduce this time when you reduce the tcp-keepalive value on your redis server

const Redis = require("ioredis");

(async () => {
  const client = new Redis({
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_PORT,
    username: process.env.REDIS_USERNAME,
    password: process.env.REDIS_PASSWORD,
  });

  client.on("error", (err) => {
    console.log("client error", new Date(), err);
  });

  const publisher = new Redis({
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_PORT,
    username: process.env.REDIS_USERNAME,
    password: process.env.REDIS_PASSWORD,
  });

  publisher.on("error", (err) =>
    console.log("publisher error", new Date(), err)
  );

  const subscriber = new Redis({
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_PORT,
    username: process.env.REDIS_USERNAME,
    password: process.env.REDIS_PASSWORD,
  });

  subscriber.on("error", (err) => {
    console.log("subscriber error", new Date(), err);
  });

  subscriber.subscribe("ioredis:msg");

  subscriber.on("message", (channel, message) => {
    console.log(new Date(), message);
  });

  setInterval(() => {
    publisher.publish("ioredis:msg", "published @ " + new Date());
  }, 2500);
})();

I noticed that the error event for the subscriber doesn't get fired every time.

subscriber.on("error", (err) => {
  console.log("subscriber error", new Date(), err);
});
If it would, I could initiate the ping from from there.

My working fix

Ping the server regularly.

setInterval(() => {
  subscriber.ping()
}, 2500);

[marcbachmann]

The issue is most likely that there's no keepAlive configured. And for the keepalive itself we have another issue open: https://github.com/luin/ioredis/issues/1339

[marcbachmann]

question is why isn't a connection abort detected? I think it should resubscribe. Sorry, will reopen that.

[marcbachmann]

Could you test that with ioredis v5? most likely we don't know about the disconnect until some I/O happens. Fixing keepalive should get that working more reliably.

[Mall0c]

Is this somehow tackled at the moment? The README says that a subscribed client will resubscribe to a channel, but for this the connection loss must be detected. The ping trick does its thing, but it would be cooler if a connection loss would be detected.

[marcbachmann]

A fix for the keepalive landed on the 31. of march https://github.com/luin/ioredis/pull/1554 Did you run into the same issue?

[Mall0c]

Ah I see. We use node.js LTS Version, which is 16.x. AFAICT, this would work with node >= 17.7.0, right?

[marcbachmann]

The applied fix in ioredis v5.0.3 should work for all the node.js versions. The mention of 17.7 was just about another new configuration option in the socket api, but that's not in use at the moment.

[marcbachmann]

Closing as we can assume the keepalive change fixed it