Connection to Redis fails after instance restart on Azure

[MichalJakubeczy]

We are using following code to connect to our caches (in-memory and Redis):

settings
  .WithSystemRuntimeCacheHandle()
  .WithExpiration(CacheManager.Core.ExpirationMode.Absolute, defaultExpiryTime)
  .And
  .WithRedisConfiguration(CacheManagerRedisConfigurationKey, connectionString)
  .WithMaxRetries(3)
  .WithRetryTimeout(100)
  .WithJsonSerializer()
  .WithRedisBackplane(CacheManagerRedisConfigurationKey)
  .WithRedisCacheHandle(CacheManagerRedisConfigurationKey, true)
  .WithExpiration(CacheManager.Core.ExpirationMode.Absolute, defaultExpiryTime);

It works fine, but sometimes machine is restarted (automatically by Azure where we host it) and after the restart connection to Redis fails with following exception:

Connection to '{connection string}' failed.
   at CacheManager.Core.BaseCacheManager`1..ctor(String name, ICacheManagerConfiguration configuration)
   at CacheManager.Core.BaseCacheManager`1..ctor(ICacheManagerConfiguration configuration)
   at CacheManager.Core.CacheFactory.Build[TCacheValue](String cacheName, Action`1 settings)
   at CacheManager.Core.CacheFactory.Build(Action`1 settings)

According to Redis FAQ (https://docs.microsoft.com/en-us/azure/redis-cache/cache-faq) part: "Why was my client disconnected from the cache?" it might happen after redeploy.

The question is

• is there any mechanism to restore the connection after redeploy
• is anything wrong in way we initialize the connection

We are sure the connection string is OK

[MichaCo]

You'll get that exception only if the Redis Server is not available on app startup.

If your app is running and Redis is down for a few seconds, the connection should recover automatically after a short delay... You might get some timeout or server exceptions obviously, until it does recover.

There are a few settings which can be passed through to the Stackexchange client, like connectionTimeout. set it to a higher value if you want to wait a little longer might make sense.

If you want to do more advanced things with the connection, you can initialize the ConnectionMultiplexer youself and hook into connect/reconnect/error events and such and handle that connection how ever you want. Simply pass in the connection multiplexer instance to CacheManager configuration / redis part instead of a connection string if you choose to do so.

Hope that helps, let me know if you have any issues

[MichaCo]

/closing as that seems to be no issue with CacheManager but the the setup/connection handling in general.

[erizzo]

I can reproduce the lack of reconnection locally.

1. I start my app, local Redis server is running.
2. Shut off my Redis server. When my app tries to write to the cache (configured with in-memory handle and Redis handle), it reports connection exceptions*.
3. Turn Redis server back on.
4. When my app tries to write to Redis, no exceptions are logged but also no data ends up in the Redis database.

Restarting the app gets things back in order, but the lack of information during calls to Put() are a problem. We're also seeing this in Azure environments (app deployed to Azure and using Azure Redis).

• Exceptions look like this: CacheManager.Redis.RedisCacheBackplane[0] Error occurred sending backplane messages. StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: PUBLISH CacheManagerBackplane; An existing connection was forcibly closed by the remote host; IOCP: (Busy=0, Free=1000,Min=10,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), Local-CPU: n/a ---> StackExchange.Redis.RedisConnectionException: SocketFailure (ReadSocketError/ConnectionReset, last-recv: 134) on localhost:6379/Subscription, Idle, last: PING, origin: ReadFromPipe, outstanding: 0, last-read: 221s ago, last-write: 40s ago, unanswered-write: 221s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 8 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.513.63329 ---> Pipelines.Sockets.Unofficial.ConnectionResetException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

[MichaCo]

When my app tries to write to Redis, no exceptions are logged but also no data ends up in the Redis database.

Ok that sounds fishy, are you really sure? I did test that a bit a long time ago and the StackExchange.Redis client was pretty good in re-connecting and stuff just continued working from then on. I have to check if there are any bigger changes in how that client behaves now, haven't really tested the latest versions

[erizzo]

Yes, I'm 100% certain of the observed behavior. I've been verifying it locally all day today. I wonder if I'm bumping against the expiration, which I explicitly set using Expire(key, DateTimeOffset) whenever I update values. I just noticed that the docs for Expire() indicate that it uses Fire-and-Forget (Put()). I'm also using Put() when inserting/updating values. Thinking out loud here... The exceptions I see when Redis is shut down are only about backplane, nothing about actually storing the value. When Redis comes back, those backplane exceptions go away, but Redis never gets an update. Could that be due to something about the 2 handles having the same expiration that I set via Expire()? Could the Put() or Expire() calls be failing at the Redis handle but just not reporting the error?