Getting intermittent errors when performing parallel gRPC calls.

[voroninp]

What version of gRPC and what language are you using?

C#

What operating system (Linux, Windows,...) and version?

Client: Windows Server 2012, Server: Docker Linux container

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

client: .Net Framework 4.6 Grpc.Core server: .NET 7

What did you do?

I have multiple parallel gRPC calls. Quite often (but not always) I see an error for some of the calls:

{  
    Type: 'System.IO.IOException',
    Data: {
        ActivityId: '00-2f69e7db02bab0398d5790f160790820-4a685c674cb0fe0b-01'
    },
    HResult: -2146232800,
    Message: 'The request stream was aborted.',
    Source: 'System.Private.CoreLib',
    TargetSite: 'Void Throw()',
    InnerException: {    
        HResult: -2146233029,
        Message: 'The HTTP/2 connection faulted.',  
        Source: null,
        CancellationToken: {      
            IsCancellationRequested: false,
            CanBeCanceled: false,      
            WaitHandle: {        
               Handle: { },        
               SafeWaitHandle: {         
                   IsInvalid: false,
                   IsClosed: false
               }
            }
        },
        Type: 'Microsoft.AspNetCore.Connections.ConnectionAbortedException'
    }
}

With stack trace:

System.IO.IOException: The request stream was aborted. ---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted. --- End of inner exception stack trace --- at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.GetReadAsyncResult() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func2 deserializer, CancellationToken cancellationToken) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at Grpc.AspNetCore.Server.Internal.HttpContextStreamReader1.<MoveNext>g__MoveNextAsync|11_0(ValueTask1 readStreamTask) at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+MoveNext() at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at DataProvider.Service.Endpoint.RadioDataProvider.RespondWithEntitiesHavingIdsFromStream[TSource,TMapped](IAsyncStreamReader1 requestStream, IAsyncStreamWriter1 responseStream, Task1 availableDataTask, Func3 mapper, CancellationToken cancellationToken) in /src/src/DataProvider.Service.Endpoint/RadioDataProvider.cs:line 473 at DataProvider.Service.Endpoint.RadioDataProvider.RespondWithEntitiesHavingIdsFromStream[TSource,TMapped](IAsyncStreamReader1 requestStream, IAsyncStreamWriter1 responseStream, Task1 availableDataTask, Func3 mapper, CancellationToken cancellationToken) in /src/src/DataProvider.Service.Endpoint/RadioDataProvider.cs:line 473 at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker3.ResolvedInterceptorInvoker(IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext resolvedContext) at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker3.ResolvedInterceptorInvoker(IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext resolvedContext) at CalculationEngine.Shared.Infrastructure.ClientInitiatedCancellationInterceptor.DuplexStreamingServerHandler[TRequest,TResponse](IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext context, DuplexStreamingServerMethod2 continuation) in /src/src/CalculationEngine.Shared.Infrastructure/ClientInitiatedCancellationInterceptor.cs:line 84 at Grpc.Shared.Server.InterceptorPipelineBuilder2.<>c__DisplayClass3_0.<<DuplexStreamingPipeline>b__1>d.MoveNext() --- End of stack trace from previous location --- at Grpc.Shared.Server.InterceptorPipelineBuilder2.<>c__DisplayClass3_0.<<DuplexStreamingPipeline>b__1>d.MoveNext() --- End of stack trace from previous location --- at CalculationEngine.Shared.Server.Grpc.StatusFromDomainExceptionAugmentingInterceptor.DuplexStreamingServerHandler[TRequest,TResponse](IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext context, DuplexStreamingServerMethod2 continuation) in /src/src/CalculationEngine.Shared.Server.Grpc/StatusFromDomainExceptionAugmentingInterceptor.cs:line 55 at Grpc.Shared.Server.InterceptorPipelineBuilder2.<>c__DisplayClass3_0.<<DuplexStreamingPipeline>b__1>d.MoveNext() --- End of stack trace from previous location --- at Grpc.Shared.Server.InterceptorPipelineBuilder2.<>c__DisplayClass3_0.<<DuplexStreamingPipeline>b__1>d.MoveNext() --- End of stack trace from previous location --- at CalculationEngine.Shared.Infrastructure.GrpcCallLoggingInterceptor.DuplexStreamingServerHandler[TRequest,TResponse](IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream, ServerCallContext context, DuplexStreamingServerMethod2 continuation) in /src/src/CalculationEngine.Shared.Infrastructure/GrpcCallLoggingInterceptor.cs:line 94 at Grpc.Shared.Server.InterceptorPipelineBuilder2.<>c__DisplayClass3_0.<<DuplexStreamingPipeline>b__1>d.MoveNext() --- End of stack trace from previous location --- at Grpc.Shared.Server.InterceptorPipelineBuilder2.<>c__DisplayClass3_0.<<DuplexStreamingPipeline>b__1>d.MoveNext() --- End of stack trace from previous location --- at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader1 requestStream, IServerStreamWriter1 responseStream) at Grpc.AspNetCore.Server.Internal.CallHandlers.DuplexStreamingServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)

What did you expect to see?

Successful calls without errors.