Redis not reconnecting after Redis::ConnectionError when testing primary failover on Elasticache

[ndbroadbent]

Hello, this is for Redis version 4.1.3.

I am trying to make sure that my application never crashes during a primary failover on Elasticache. I added this to my Redis options (via Sidekiq):

  reconnect_attempts: 7,
  reconnect_delay: 1.0,
  reconnect_delay_max: 5.0,

I deployed these changes (to staging), and then tested the "Failover primary" in the AWS console:

I was expecting my API requests to pause for a few seconds while Redis automatically reconnects to the new primary. However, the request paused and then crashed with:

Redis::CannotConnectError
Error connecting to Redis on *******.******.ng.0001.use1.cache.amazonaws.com:6379 (Redis::TimeoutError)

Is there any way I can fix this to automatically reconnect during an ElastiCache auto-failover?

---

Full stack trace for the exception:

IO::EINPROGRESSWaitWritable: Operation now in progress - connect(2) would block
  from socket.rb:1213:in `__connect_nonblock'
  from socket.rb:1213:in `connect_nonblock'
  from redis/connection/ruby.rb:180:in `connect_addrinfo'
  from redis/connection/ruby.rb:220:in `block in connect'
  from redis/connection/ruby.rb:218:in `each'
  from redis/connection/ruby.rb:218:in `each_with_index'
  from redis/connection/ruby.rb:218:in `connect'
  from redis/connection/ruby.rb:317:in `connect'
  from redis/client.rb:349:in `establish_connection'
  from redis/client.rb:106:in `block in connect'
  from redis/client.rb:306:in `with_reconnect'
  from redis/client.rb:105:in `connect'
  from redis/client.rb:381:in `ensure_connected'
  from redis/client.rb:231:in `block in process'
  from redis/client.rb:319:in `logging'
  from redis/client.rb:230:in `process'
  from redis/client.rb:125:in `call'
  from redis.rb:1956:in `block in zrangebyscore'
  from redis.rb:52:in `block in synchronize'
  from monitor.rb:235:in `mon_synchronize'
  from redis.rb:52:in `synchronize'
  from redis.rb:1955:in `zrangebyscore'
  from sidekiq/scheduled.rb:20:in `block (2 levels) in enqueue_jobs'
  from sidekiq/scheduled.rb:15:in `each'
  from sidekiq/scheduled.rb:15:in `block in enqueue_jobs'
  from sidekiq.rb:97:in `block in redis'
  from connection_pool.rb:65:in `block (2 levels) in with'
  from connection_pool.rb:64:in `handle_interrupt'
  from connection_pool.rb:64:in `block in with'
  from connection_pool.rb:61:in `handle_interrupt'
  from connection_pool.rb:61:in `with'
  from sidekiq.rb:94:in `redis'
  from sidekiq/scheduled.rb:14:in `enqueue_jobs'
  from sidekiq/scheduled.rb:77:in `enqueue'
  from sidekiq/scheduled.rb:68:in `block in start'
  from sidekiq/util.rb:16:in `watchdog'
  from sidekiq/util.rb:25:in `block in safe_thread'
Redis::TimeoutError: Redis::TimeoutError
  from redis/connection/ruby.rb:183:in `rescue in connect_addrinfo'
  from redis/connection/ruby.rb:179:in `connect_addrinfo'
  from redis/connection/ruby.rb:220:in `block in connect'
  from redis/connection/ruby.rb:218:in `each'
  from redis/connection/ruby.rb:218:in `each_with_index'
  from redis/connection/ruby.rb:218:in `connect'
  from redis/connection/ruby.rb:317:in `connect'
  from redis/client.rb:349:in `establish_connection'
  from redis/client.rb:106:in `block in connect'
  from redis/client.rb:306:in `with_reconnect'
  from redis/client.rb:105:in `connect'
  from redis/client.rb:381:in `ensure_connected'
  from redis/client.rb:231:in `block in process'
  from redis/client.rb:319:in `logging'
  from redis/client.rb:230:in `process'
  from redis/client.rb:125:in `call'
  from redis.rb:1956:in `block in zrangebyscore'
  from redis.rb:52:in `block in synchronize'
  from monitor.rb:235:in `mon_synchronize'
  from redis.rb:52:in `synchronize'
  from redis.rb:1955:in `zrangebyscore'
  from sidekiq/scheduled.rb:20:in `block (2 levels) in enqueue_jobs'
  from sidekiq/scheduled.rb:15:in `each'
  from sidekiq/scheduled.rb:15:in `block in enqueue_jobs'
  from sidekiq.rb:97:in `block in redis'
  from connection_pool.rb:65:in `block (2 levels) in with'
  from connection_pool.rb:64:in `handle_interrupt'
  from connection_pool.rb:64:in `block in with'
  from connection_pool.rb:61:in `handle_interrupt'
  from connection_pool.rb:61:in `with'
  from sidekiq.rb:94:in `redis'
  from sidekiq/scheduled.rb:14:in `enqueue_jobs'
  from sidekiq/scheduled.rb:77:in `enqueue'
  from sidekiq/scheduled.rb:68:in `block in start'
  from sidekiq/util.rb:16:in `watchdog'
  from sidekiq/util.rb:25:in `block in safe_thread'
Redis::CannotConnectError: Error connecting to Redis on ********.*****.ng.0001.use1.cache.amazonaws.com:6379 (Redis::TimeoutError)
  from redis/client.rb:362:in `rescue in establish_connection'
  from redis/client.rb:343:in `establish_connection'
  from redis/client.rb:106:in `block in connect'
  from redis/client.rb:306:in `with_reconnect'
  from redis/client.rb:105:in `connect'
  from redis/client.rb:381:in `ensure_connected'
  from redis/client.rb:231:in `block in process'
  from redis/client.rb:319:in `logging'
  from redis/client.rb:230:in `process'
  from redis/client.rb:125:in `call'
  from redis.rb:1956:in `block in zrangebyscore'
  from redis.rb:52:in `block in synchronize'
  from monitor.rb:235:in `mon_synchronize'
  from redis.rb:52:in `synchronize'
  from redis.rb:1955:in `zrangebyscore'
  from sidekiq/scheduled.rb:20:in `block (2 levels) in enqueue_jobs'
  from sidekiq/scheduled.rb:15:in `each'
  from sidekiq/scheduled.rb:15:in `block in enqueue_jobs'
  from sidekiq.rb:97:in `block in redis'
  from connection_pool.rb:65:in `block (2 levels) in with'
  from connection_pool.rb:64:in `handle_interrupt'
  from connection_pool.rb:64:in `block in with'
  from connection_pool.rb:61:in `handle_interrupt'
  from connection_pool.rb:61:in `with'
  from sidekiq.rb:94:in `redis'
  from sidekiq/scheduled.rb:14:in `enqueue_jobs'
  from sidekiq/scheduled.rb:77:in `enqueue'
  from sidekiq/scheduled.rb:68:in `block in start'
  from sidekiq/util.rb:16:in `watchdog'
  from sidekiq/util.rb:25:in `block in safe_thread'