node-redis icon indicating copy to clipboard operation
node-redis copied to clipboard

AbortError: Redis connection lost and command aborted. It might have been processed.

Open theromie opened this issue 4 years ago • 7 comments

I have 2 server one is for API and one is for Redis and I am connecting the API server to the Redis server using Redis VMs IP address and on the API server I ran the pm2 command to start the API server and for 300-sec Redis server is alive and when API try to fetch data from Redis server I am getting - AbortError: Redis connection lost and command aborted. It might have been processed. with econnreset. How can I keep the connection alive? I am using the Redis version on server Redis server v=4.0.9

theromie avatar Oct 16 '20 12:10 theromie

const client = redis.createClient({
  port: 6379,
  host: REDIS_HOSTNAME,
  password: REDIS_PASSWORD,
  retry_strategy: function (options) {
    if (options.error && options.error.code === "ECONNREFUSED") {
      // End reconnecting on a specific error and flush all commands with
      // a individual error
      return new Error("The server refused the connection");
    }
    if (options.total_retry_time > 1000 * 60 * 60) {
      // End reconnecting after a specific timeout and flush all commands
      // with a individual error
      return new Error("Retry time exhausted");
    }
    if (options.attempt > 10) {
      // End reconnecting with built in error
      return undefined;
    }
    // reconnect after
    return Math.min(options.attempt * 100, 3000);
  },
  max_attempts: 1,
});

client.on('connect', () => {
  console.log('Client connected to redis...');
});

client.on('ready', () => {
  console.log('Client connected to redis and ready to use...');
});
client.KEYS("*", (err, res) => {
  console.log(res);
});

client.on('reconnecting', () => {
  console.log('Client reconnecting...');
});

client.on('error', (err) => {
  console.log({ err });
});

client.on('end', () => {
  console.log('Client disconnected from redis');
});

process.on('SIGINT', () => {
  client.quit();
});

This my code

theromie avatar Oct 16 '20 12:10 theromie

I also encountered this problem. If retry_strategy is set, then more than 10 reconnections fail, and then no reconnection attempts will be made. I currently choose not to set retry_strategy, so node-redis will always try to reconnect.

Hope there is a better solution

MalavitaC avatar Apr 23 '21 04:04 MalavitaC

Any progress on this issue ? I have not set any retry strategy I am still encountering this issue .

nambirajan-goxai avatar Jul 05 '22 04:07 nambirajan-goxai

When a socket closes unexpectedly, all the commands that were already sent (written on the socket) will reject as they might have been executed on the server. The rest will remain queued in memory until a new socket is established.

leibale avatar Jul 05 '22 15:07 leibale

Try this code. On ECONNREFUSED you need to return a number to retry.

const client = require('redis').createClient(process.env.REDIS_HOST, {
    retry_strategy: function(options) {
        
        if (options.error && options.error.code === "ECONNREFUSED") {
            // End reconnecting on a specific error and flush all commands with
            // a individual error
           console.log("The server refused the connection")

            return 1000 // <- This retries the connection instead of crashing server
        }
        if (options.total_retry_time > 1000 * 60 * 60) {
            // End reconnecting after a specific timeout and flush all commands
            // with a individual error
            console.log("Retry time exhausted")
            return new Error("Retry time exhausted");
        }
        if (options.attempt > 10) {
            // End reconnecting with built in error
            return undefined;
        }
        // reconnect after
        return Math.min(options.attempt * 100, 3000);
    }
});

leovazquezz1 avatar Jul 18 '22 14:07 leovazquezz1

Hey @leovazquezz1, that can cause an infinite loop if the connection is refused for something that won't resolve itself (a firewall blocking the calls, for example).

Removing the return 1000 will fix that. It means the following 2 if conditions will break any potential infinite loop and the Math.min(options.attempt * 100, 3000) will return the time to wait until next retry, incrementing the wait period as expected.

scottjferguson avatar Jul 19 '22 18:07 scottjferguson

I was able to find a workaround for these errors with the code below. Basically, the return in the retryOptions.error.code === 'ECONNREFUSED' condition was causing 'connection refused' errors to be treated as an Error the first time it was encountered, and blocked this type of error from being retried. Since applying this fix, I've run 2 load tests simulating 200 users without seeing the error at all.

So it doesn't address the connection error directly, but it will put your code in a state where it can recover from the error and try again:

const client = redis.createClient({
    host: process.env.REDISHOST,
    port: process.env.REDISPORT,
    retry_strategy: (retryOptions) => {
        if (retryOptions.error && retryOptions.error.code === 'ECONNREFUSED') {
            console.log('CacheStore - The server refused the connection');
            // *** Removing the return statement here was key. It was blocking the retry strategy from executing. ***
        }

        if (retryOptions.total_retry_time > 1000 * 60 * 60) {
            console.log('CacheStore retry strategy was not successful. Retry time exceeded.');
            // End reconnecting after a specific timeout and flush all commands with a individual error
            return new Error('Retry time exhausted');
        }

        if (retryOptions.attempt > 10) {
            console.log('CacheStore retry strategy was not successful. Retry attempt count exceeded.');
            // End reconnecting with built in error
            return undefined;
        }

        // Reconnect after
        return Math.min(options.attempt * 100, 3000);
    }

scottjferguson avatar Jul 21 '22 16:07 scottjferguson