Thread pool starvation caused by ConnectionMultiplexer.UpdateSentinelAddressList(string serviceName = "sentinel")

[napernik]

Good day.

We have encountered an issue where a lot of the threads (~900) were blocked by the code that's supposed to connect to Redis Sentinel instances:

System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout = -1, System.Threading.CancellationToken cancellationToken) Line 277	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2169	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler scheduler, bool waitForCompletion) Line 1413	C#
System.Linq.Parallel.dll!System.Linq.Parallel.SpoolingTask.SpoolStopAndGo<System.Net.EndPoint[], int>(System.Linq.Parallel.QueryTaskGroupState groupState, System.Linq.Parallel.PartitionedStream<System.Net.EndPoint[], int> partitions, System.Linq.Parallel.SynchronousChannel<System.Net.EndPoint[]>[] channels, System.Threading.Tasks.TaskScheduler taskScheduler)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.DefaultMergeHelper<System.Net.EndPoint[], int>.System.Linq.Parallel.IMergeHelper<System.Net.EndPoint[]>.Execute()	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<System.Net.EndPoint[]>.Execute<int>(System.Linq.Parallel.PartitionedStream<System.Net.EndPoint[], int> partitions, bool ignoreOutput, System.Linq.ParallelMergeOptions options, System.Threading.Tasks.TaskScheduler taskScheduler, bool isOrdered, System.Linq.Parallel.CancellationState cancellationState, int queryId)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.PartitionedStreamMerger<System.Net.EndPoint[]>.Receive<int>(System.Linq.Parallel.PartitionedStream<System.Net.EndPoint[], int> partitionedStream)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<System.Net.EndPoint[]>.WrapHelper<int>(System.Linq.Parallel.PartitionedStream<System.Net.EndPoint[], int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<System.Net.EndPoint[]> recipient, System.Linq.Parallel.QuerySettings settings)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<System.Net.EndPoint[]>.WrapPartitionedStream<int>(System.Linq.Parallel.PartitionedStream<System.Net.EndPoint[], int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<System.Net.EndPoint[]> recipient, bool preferStriping, System.Linq.Parallel.QuerySettings settings)	Unknown
[Native to Managed Transition]	
[Managed to Native Transition]	
System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<System.Net.EndPoint[], System.Net.EndPoint[]>.UnaryQueryOperatorResults.ChildResultsRecipient.Receive<int>(System.Linq.Parallel.PartitionedStream<System.Net.EndPoint[], int> inputStream)	Unknown
[Native to Managed Transition]	
[Managed to Native Transition]	
System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<StackExchange.Redis.ServerEndPoint, System.Net.EndPoint[]>.UnaryQueryOperatorResults.ChildResultsRecipient.Receive<int>(System.Linq.Parallel.PartitionedStream<StackExchange.Redis.ServerEndPoint, int> inputStream)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.ScanQueryOperator<StackExchange.Redis.ServerEndPoint>.ScanEnumerableQueryOperatorResults.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<StackExchange.Redis.ServerEndPoint> recipient)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<StackExchange.Redis.ServerEndPoint, System.Net.EndPoint[]>.UnaryQueryOperatorResults.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<System.Net.EndPoint[]> recipient)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<System.Net.EndPoint[], System.Net.EndPoint[]>.UnaryQueryOperatorResults.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<System.Net.EndPoint[]> recipient)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.QueryOperator<System.Net.EndPoint[]>.GetOpenedEnumerator(System.Linq.ParallelMergeOptions? mergeOptions, bool suppressOrder, bool forEffect, System.Linq.Parallel.QuerySettings querySettings)	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<System.Net.EndPoint[]>.OpenQuery()	Unknown
System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<System.Net.EndPoint[]>.MoveNext()	Unknown
System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.GetOneWithPossibleDefault<System.Net.EndPoint[]>(System.Linq.Parallel.QueryOperator<System.Net.EndPoint[]> queryOp, bool throwIfTwo = false, bool defaultIfEmpty = true)	Unknown
System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.FirstOrDefault<System.Net.EndPoint[]>(System.Linq.ParallelQuery<System.Net.EndPoint[]> source, System.Func<System.Net.EndPoint[], bool> predicate)	Unknown
StackExchange.Redis.dll!StackExchange.Redis.ConnectionMultiplexer.UpdateSentinelAddressList(string serviceName = "sentinel") Line 423	C#
StackExchange.Redis.dll!StackExchange.Redis.ConnectionMultiplexer.InitializeSentinel.AnonymousMethod__2(StackExchange.Redis.RedisChannel _, StackExchange.Redis.RedisValue message) Line 77	C#
StackExchange.Redis.dll!StackExchange.Redis.MessageCompletable.TryComplete(bool isAsync) Line 32	C#
StackExchange.Redis.dll!StackExchange.Redis.ConnectionMultiplexer..cctor.AnonymousMethod__77_0(object s) Line 9	C#
System.Private.CoreLib.dll!System.Threading.QueueUserWorkItemCallback.Execute() Line 28	C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 487	C#

Link to the related source code: https://github.com/StackExchange/StackExchange.Redis/blob/2.7.33/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs#L423,L431

It looks like this code allocates all of the threads because of AsParallel() call, and there's no free thread to unblock itself.

I've also check the memory dump for exceptions, and there's one Redis timeout exception:

And one exception saying "s_doneWritingSentinel"

StackExchange.Redis version: 2.7.33

Please advice if it is a known issue, and if there are any workarounds.