ConnectTimeout when sending multiple GET commands immediately after service start

[incloudss]

Hi,

I am getting exception: It was not possible to connect to the redis server(s). ConnectTimeout" when sending ~20 GET commands immediately after service start, parallalely(each GET is invoked from different thread). I am sure i have singleton instance of RedisCacheClient, as i am registering it through StackExchange.Redis.Extensions.AspNetCore package.

The workaround i found for now is to send one GET command after starting service, from main thread, and then proceed to sending multiple requests paralelly, and it works flawlessly.

Full exception stacktrace

at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1032
at StackExchange.Redis.Extensions.Core.Implementations.RedisCacheConnectionPoolManager.<EmitConnection>b__10_0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Lazy`1.CreateValue()
at StackExchange.Redis.Extensions.Core.Implementations.RedisCacheConnectionPoolManager.GetConnection()
at StackExchange.Redis.Extensions.Core.Implementations.RedisDatabase.SubscribeAsync[T](RedisChannel channel, Func`2 handler, CommandFlags flags)
at MotoGo.ExpressBus.ExpressBus.<Receive>d__9`1.MoveNext() in D:\Repos\MotoGo\src\MotoGo.ExpressBus\ExpressBus.cs:line 120
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
--- End of stack trace from previous location ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
--- End of stack trace from previous location ---
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MotoGo.CacheLoader.HostedService.<StartAsync>d__10.MoveNext() in D:\Repos\MotoGo\src\MotoGo.CacheLoader\HostedService.cs:line 54```
</details>

[OriginalArkus]

Me too! I thought this was my Docker environment being silly, but turns out I'm not the only person.

Watching this.

[NickCraver]

Given the symptoms, you're likely hitting threadpool exhaustion overall in the app, early, before it grows. As an experiment (bain-aid, not a real fix) you can try ThreadPool.SetMinThreads() before the connection runs to see how it behaves. There's no magic bullet here because it'll depend on your environment, budget, how many cores are available, what work is competing, etc. But if you're quickly going full parallel and synchronously so when the app starts, you may be exhausting the available thread pool very rapidly.

Generally, you want to go async here to schedule work and not tie up threads with waiting and inaction, but that may be a non-trivial change...it is the best long term play though :)

[garpunkal]

We're experiencing the same issue with .net 4.6.2.

[NickCraver]

Closing this out to tidy up - best advice we have is above!