node-redis icon indicating copy to clipboard operation
node-redis copied to clipboard

cluster slots not rediscovered on scale-down cluster

Open carlhopf opened this issue 1 year ago • 3 comments

Description

#slots.rediscover() is correctly called when MOVED or ASK reply is received:

https://github.com/redis/node-redis/blob/6f79b49f731a1aaf57f42e16ea72774f062e57a1/packages/client/lib/cluster/index.ts#L271

https://github.com/redis/node-redis/blob/6f79b49f731a1aaf57f42e16ea72774f062e57a1/packages/client/lib/cluster/index.ts#L256

there is a problem / race-condition on scale-down cluster (rebalance hash slots away, forget node, shutdown):

#slots.rediscover() might never be called, if the leaving node has not been queried (after it's slots migrated away) before shutdown (hence it never got a chance to reply with MOVED and trigger a rediscover)

this results in connection closed errors, because node-redis cluster.#slots is keeping a client active for an outdated cluster topology.

steps to reproduce:

  1. start a redis cluster
  2. create node-redis client and connect
  3. redis-cli reshard all slots away from a redis cluster node
  4. redis-cli del-node the now empty node, then shutdown
  5. node-redis client execute command for a key, which is sharded the (now shutdown) node
  6. connection closed error

an option to fix this would be to provide an option for a slot refresh interval (ioredis does this too), will send a pull request

Node.js Version

No response

Redis Server Version

No response

Node Redis Version

No response

Platform

No response

Logs

No response

carlhopf avatar Jul 26 '24 11:07 carlhopf