ioredis icon indicating copy to clipboard operation
ioredis copied to clipboard

4.27.0 introduced a change that prevents the event loop from being cleaned up properly

Open evanlucas opened this issue 3 years ago • 8 comments

https://github.com/evanlucas/ioredis-sentinel-hang is how I was able to reproduce it. We rely on calling quit in all of our jobs/workers/services and if the event loop is not cleaned up, it prevents node from cleanly exiting the process. It looks like it is the new failover detection mechanism that was added. I'm seeing additional TCP sockets being opened, but never closed.

This only seems to occur when using redis sentinel.

Thanks!

evanlucas avatar Jun 03 '21 19:06 evanlucas

confirming, it opens a bunch of new connections that are never cleaned up (see FailoverDetector)

AVVS avatar Jun 04 '21 22:06 AVVS

Hey @evanlucas, thanks for the reproducible repo! Calling .disconnect() should properly clean up everything. We should take a look for a fix for quit.

luin avatar Jun 05 '21 07:06 luin

Thanks for jumping on this and getting a non-breaking release published! I really appreciate y'all's time! Did y'all want to leave this issue open? or is it good to close?

evanlucas avatar Jun 14 '21 11:06 evanlucas

Hey @evanlucas, let's leave this open as the root issue hasn't been fixed.

luin avatar Jun 14 '21 11:06 luin

Any progress on this one. It would be fantastic to get this one resolved

esatterwhite avatar Oct 05 '21 14:10 esatterwhite

In a Kubernetes cluster, when one of the Sentinels restarts and gets a new IP address, client starts throwing errors like this one:

[ioredis] Unhandled error event: Error: connect EHOSTUNREACH 10.0.55.221:26379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17)

Where 10.0.55.221 is the old Sentinel IP address.

Issue disappears with failoverDetector: false

juris avatar Feb 07 '22 09:02 juris

AFAICT failoverDetector: true is a useful feature. I am testing Redis Sentinel on a local setup with three VMs, each of them with a Redis and a Redis Sentinel instance. Shutting down the VM that holds the Redis Master ungracefully with Force Off in the Virtual Machine Manager does not trigger a Reconnect. Somehow ioredis does not that Sentinel has performed a failover, unless this flag is set to true. For now, I went with calling quit() and disconnect() after that. Seems to work.

Mall0c avatar Jun 08 '22 13:06 Mall0c

There's definitely something fishy going on when failoverDetector is enabled. After a failover, ioredis will start logging Unhandled error event to stderr, despite me having registered the error event earlier which should prevent all such logging.

silverwind avatar Sep 19 '22 16:09 silverwind