Unable to lock multi resources in cluster mode

[Simon-CN]

node-relock version: 4.1.0 ioredis version: 4.14.1

Hi, I'm trying to use node-redlock with redis-cluster. When locking multi resources, I got a "LockError" exception. It exceeds the 10 attempts to lock.

Here's my code

  let client = new redis.Cluster([
        {
            ip: '127.0.0.1',
            port: '7001'
        },
        {
            ip: '127.0.0.1',
            port: '7002'
        },
        {
            ip: '127.0.0.1',
            port: '7003'
        }
    ])
    let lk = new redlock([client])
    let l = null
    l = await lk.lock(['lk1', 'lk2'], 1000) //failure
    //l = await lk.lock(['lk'], 1000)  //success
    let res = await client.get('foo')

(node:28707) UnhandledPromiseRejectionWarning: LockError: Exceeded 10 attempts to lock the resource "lk1,lk2".
    at /home/sx/projects/nodejs/njproj1/node_modules/redlock/redlock.js:411:20
    at tryCatcher (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.errorAdapter [as _rejectionHandler0] (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/nodeify.js:35:34)
    at Promise._settlePromise (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:601:21)
    at Promise._settlePromise0 (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:15:14)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)

[richard-julien]

Hi @mike-marcacci ,

Just tested this use case with the 5.0-alpha on a redis cluster (using https://github.com/bitnami/bitnami-docker-redis-cluster) with the exact same problem. Does it require to configure something specific? Thanks in advance

SUCCESS: redlock.using(['id1'], maxTtl, async (signal) => {})

FAILURE: redlock.using(['id1', 'id2'], maxTtl, async (signal) => {})

ExecutionError: The operation was unable to acheive a quorum during its retry window.
    at Redlock._execute (C:\Users\richa\Documents\Projects\opencti\opencti-platform\opencti-graphql\node_modules\redlock\src\index.ts:450:15)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Redlock.acquire (C:\Users\richa\Documents\Projects\opencti\opencti-platform\opencti-graphql\node_modules\redlock\src\index.ts:305:28)
    at Redlock.using (C:\Users\richa\Documents\Projects\opencti\opencti-platform\opencti-graphql\node_modules\redlock\src\index.ts:718:16)

[mike-marcacci]

Hi folks! I'm going to go ahead and close this, as I just published v5.0.0-beta.1 which includes extensive cluster tests and some small improvements over the v5 alpha that should aid in debugging.

@richard-julien consider adopting using following to extract the underlying issues from an ExecutionError:

https://github.com/mike-marcacci/node-redlock/blob/078c7270a48596db8a23b4884f88db1f1eb3c18b/src/single.test.ts#L14-L31

[joeyaurel]

Issue still is a thing in v5.0.0-beta.1. Just had the error The operation was unable to acheive a quorum during its retry window, when calling acquire.

[mike-marcacci]

Thanks for that confirmation @nickreynke. Reopening this issue to continue tracking.

When adding cluster tests I realized that there was some nuance around hash slots and added some additional notes about use with cluster/sentinel setups to the readme. Could you look over those and see if they solve your problem?

[Nau077]

the problem is still here

redlock v5.0.0-beta.1 single mode when try simply:

` const lock = await RedisHelper.$redlock.acquire([key], 2000);

try {
  await RedisHelper.$client.set(key, JSON.stringify(message));

  await lock.release();

} catch (error) {
  await lock.release();
}

`

ExecutionError: The operation was unable to acheive a quorum during its retry window. at Redlock._execute

[Ligengxin96]

Hi @mike-marcacci, same issue with Nau077

node version: v16.13.1
os: windows 10

package.json
"redis": "^4.0.1", "ioredis": "^4.28.2", "redlock": "^5.0.0-beta.1",

Here is my code

const { default: Redlock } = require("redlock");
const Client = require('ioredis');

const client = new Client();
const redlock = new Redlock(
  [client],
  {
    driftFactor: 0.01,
    retryCount: 5,
    retryDelay: 1000,
    retryJitter: 1000
  }
);

redlock.on('clientError', function(err) {
  console.error('A redis error has occurred:', err);
});

const sleep = (time) => {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve();
    }, time)
  });
}

const main = async (key) => {
  const lock = await redlock.acquire([key], 6000);
  try {
    console.log(new Date(), 'client1 do something');
    await sleep(1000);
    
    console.log(new Date(), 'client1 do something else');
    await sleep(2000);
  } catch (err) {
    console.log(err);
  } finally {
    await lock.release();
    console.log(new Date(), 'client1 release lock');
  }
}

main('key');
main('key');
main('key');

Error info:

C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:288
                throw new ExecutionError("The operation was unable to acheive a quorum during its retry window.", attempts);
                      ^

ExecutionError: The operation was unable to acheive a quorum during its retry window.
    at Redlock._execute (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:288:23)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Redlock.acquire (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:210:34)
    at async main (C:\Users\Mr.Li\MQ\client\client1.js:28:16) {        
  attempts: [
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    }
  ]
}

[Ligengxin96]

Hi @mike-marcacci, same issue with Nau077

node version: v16.13.1 os: windows 10

package.json "redis": "^4.0.1", "ioredis": "^4.28.2", "redlock": "^5.0.0-beta.1",

Here is my code

const { default: Redlock } = require("redlock");
const Client = require('ioredis');

const client = new Client();
const redlock = new Redlock(
  [client],
  {
    driftFactor: 0.01,
    retryCount: 5,
    retryDelay: 1000,
    retryJitter: 1000
  }
);

redlock.on('clientError', function(err) {
  console.error('A redis error has occurred:', err);
});

const sleep = (time) => {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve();
    }, time)
  });
}

const main = async (key) => {
  const lock = await redlock.acquire([key], 6000);
  try {
    console.log(new Date(), 'client1 do something');
    await sleep(1000);
    
    console.log(new Date(), 'client1 do something else');
    await sleep(2000);
  } catch (err) {
    console.log(err);
  } finally {
    await lock.release();
    console.log(new Date(), 'client1 release lock');
  }
}

main('key');
main('key');
main('key');

Error info:

C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:288
                throw new ExecutionError("The operation was unable to acheive a quorum during its retry window.", attempts);
                      ^

ExecutionError: The operation was unable to acheive a quorum during its retry window.
    at Redlock._execute (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:288:23)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Redlock.acquire (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:210:34)
    at async main (C:\Users\Mr.Li\MQ\client\client1.js:28:16) {        
  attempts: [
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    },
    Promise {
      {
        membershipSize: 1,
        quorumSize: 1,
        votesFor: Set(0) {},
        votesAgainst: Map(1) {
          Redis {
            options: [Object],
            _events: [Object: null prototype] {},
            _eventsCount: 0,
            _maxListeners: undefined,
            scriptsSet: {},
            addedBuiltinSet: Set(0) {},
            commandQueue: [Denque],
            offlineQueue: [Denque],
            connectionEpoch: 1,
            connector: [StandaloneConnector],
            retryAttempts: 0,
            _addedScriptHashes: {},
            _autoPipelines: Map(0) {},
            _runningAutoPipelines: Set(0) {},
            _addedScriptHashesCleanInterval: [Timeout],
            status: 'ready',
            condition: [Object],
            stream: [Socket],
            serverInfo: [Object],
            [Symbol(kCapture)]: false
          } => ResourceLockedError: The operation was applied to: 0 of 
the 1 requested resources.
              at Redlock._attemptOperationOnClient (C:\Users\Mr.Li\MQ\node_modules\redlock\dist\cjs\index.js:379:23)
              at processTicksAndRejections (node:internal/process/task_queues:96:5)
        }
      }
    }
  ]
}

I try to use "redlock": "^4.2.0", The same code get different error: LockError: Exceeded 5 attempts to lock the resource "key". I think I know what happened. Does this error meaning still can't get lock after max retry count. @mike-marcacci

[RuoFeng2015]

出现了同样的问题 The operation was unable to acheive a quorum during its retry window

[shonmorgun]

This is still an issue guys

[Prabh-Kang]

@RuoFeng2015 @shonmorgun . I faced the same issue and going through the documentation revealed that the lock can only be applied to keys which have not been set yet. If you are trying to lock a key which is already redis db, then the lock execution will always fail. Try to run the same code with a new key.

You can find the info in attached screenshot on documentation for 4.2.0 on npmjs

[taco-wang]

hi guys， i have find the way to solve this problem. if fact , this error occur when the retry time access the max times in the init config. the source code is

if (attempts.length < maxAttempts) {
                await new Promise((resolve) => {
                    setTimeout(resolve, Math.max(0, settings.retryDelay +
                        Math.floor((Math.random() * 2 - 1) * settings.retryJitter)), undefined);
                });
            }
            else {
                throw new ExecutionError("The operation was unable to achieve a quorum during its retry window.", attempts);
            }

so I config the retryCount to 20 and retryDelay to 2000ms. in most of case , it did well

[shonmorgun]

Hey guys. I was able to workaround this issue by running my instance in WSL (Windows Subsystem for Linux). I haven't figured out why redlock is not working on Windows but it will work on Linux, and WSL. Best of luck!

[ace-cooper]

Hey there! I faced the same issue on Windows and solved it after updating ioredis to v5.0.4.

Ps: Also make sure don't use redis lib instead of ioredis.

[shonmorgun]

Hey there! I finally found the root cause of this error: The operation was unable to achieve a quorum during its retry window. It's because I had 2 redis servers running simultaneously, one auto-starts on Windows and another one running on Docker. So please make sure there's only 1 redis server running on your OS.

Cheers!

[paulohlips]

Hi folks, something new about "The operation was unable to achieve a quorum during its retry window" ?

I have the same problem using: "ioredis": "^5.2.3", "redlock": "^5.0.0-beta.2"

[simllll]

We are also facing the issue, we are using the aws elasticcache (redis) service with a 1 repli - primary setup. the primary didn't switch during the issues, so I guess this is unrelated. Maybe relevant: the unlock/release of the lock also fails sometimes?