ioredis
ioredis copied to clipboard
4.27.0 introduced a change that prevents the event loop from being cleaned up properly
https://github.com/evanlucas/ioredis-sentinel-hang is how I was able to reproduce it. We rely on calling quit
in all of our jobs/workers/services and if the event loop is not cleaned up, it prevents node from cleanly exiting the process. It looks like it is the new failover detection mechanism that was added. I'm seeing additional TCP sockets being opened, but never closed.
This only seems to occur when using redis sentinel.
Thanks!
confirming, it opens a bunch of new connections that are never cleaned up (see FailoverDetector)
Hey @evanlucas, thanks for the reproducible repo! Calling .disconnect()
should properly clean up everything. We should take a look for a fix for quit
.
Thanks for jumping on this and getting a non-breaking release published! I really appreciate y'all's time! Did y'all want to leave this issue open? or is it good to close?
Hey @evanlucas, let's leave this open as the root issue hasn't been fixed.
Any progress on this one. It would be fantastic to get this one resolved
In a Kubernetes cluster, when one of the Sentinels restarts and gets a new IP address, client starts throwing errors like this one:
[ioredis] Unhandled error event: Error: connect EHOSTUNREACH 10.0.55.221:26379
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17)
Where 10.0.55.221
is the old Sentinel IP address.
Issue disappears with failoverDetector: false
AFAICT failoverDetector: true
is a useful feature. I am testing Redis Sentinel on a local setup with three VMs, each of them with a Redis and a Redis Sentinel instance. Shutting down the VM that holds the Redis Master ungracefully with Force Off
in the Virtual Machine Manager does not trigger a Reconnect. Somehow ioredis does not that Sentinel has performed a failover, unless this flag is set to true. For now, I went with calling quit()
and disconnect()
after that. Seems to work.
There's definitely something fishy going on when failoverDetector
is enabled. After a failover, ioredis will start logging Unhandled error event
to stderr, despite me having registered the error
event earlier which should prevent all such logging.