Unhandled exception in gRPC when built using exe with dotnet 6 and TargetFramework=netcoreapp3.1

[AaronFriel]

Hey folks, I'm an engineer on the open source platform team at Pulumi and trying to debug an issue that's a little outside my wheelhouse.

We have folks using .NET 3.1 and .NET 6 CLIs in production, and we're trying to ensure our tools support both at least until (and perhaps a little after) the EOL of .NET 3.1.

I have this matrix of tests:

dotnet --version	csproj TargetFramework	Test Outcome
3.1.422	netcoreapp3.1	✅
3.1.422	net6.0	❌ (expected)
6.0.400	netcoreapp3.1	❌ gRPC unhandled exception
6.0.400	net6.0	✅

We want to support our users to be able to smoothly upgrade to dotnet 6 as 3.1 goes EOL, so I'm wondering: what could be different about this library that would cause an unhandled exception with this pairing of versions?

I think the answer is SUPPORT_LOAD_BALANCING. It's off when targeting netcoreapp3.1, but when the runtime is 6.0 it appears to crash. Any ideas on what could cause that?

Here's the exception we logged:

Unhandled exception. Pulumi.LogException: Error occurred during logging
 ---> Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: Unable to read data from the transport connection: Connection reset by peer. SocketException: Connection reset by peer", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")
   at Pulumi.GrpcEngine.LogAsync(LogRequest request)
   at Pulumi.Deployment.EngineLogger.LogAsync(LogSeverity severity, String message, Resource resource, Nullable`1 streamId, Nullable`1 ephemeral)
   --- End of inner exception stack trace ---
   at Pulumi.Deployment.EngineLogger.LogAsync(LogSeverity severity, String message, Resource resource, Nullable`1 streamId, Nullable`1 ephemeral)
   at Pulumi.Deployment.Runner.LogExceptionToErrorStream(Exception exception)
   at Pulumi.Deployment.Runner.HandleExceptionsAsync(IEnumerable`1 exceptions)
   at Pulumi.Deployment.Runner.WhileRunningAsync()
   at Pulumi.Deployment.CreateRunnerAndRunAsync(Func`1 deploymentFactory, Func`2 runAsync)
   at Program.<Main>(String[] args)

[JamesNK]

Is the address http or https? You might have this issue:

https://docs.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-6.0#call-insecure-grpc-services-with-net-core-client

Do you know exactly what assembly is being used? SUPPORT_LOAD_BALANCING isn't present at all in the NuGet package when the target is netcoreapp3.1.

[AaronFriel]

The address should be http, and I believe we are setting this switch in the two gRPC services we set up in .NET:

https://github.com/pulumi/pulumi/blob/5e570a4736e216d13b395e2524ae633e44e28a6a/sdk/dotnet/Pulumi/Deployment/GrpcEngine.cs#L20-L22

https://github.com/pulumi/pulumi/blob/5e570a4736e216d13b395e2524ae633e44e28a6a/sdk/dotnet/Pulumi/Deployment/GrpcMonitor.cs#L22-L24

I suspect then there's some sort of compat issue in a netcoreapp3.1 binary built via dotnet 6, which doesn't appear when built on dotnet 3.1.

Any idea what that could be?

[JamesNK]

If your app is using netcoreapp3.1 binary of Grpc.Net.Client on .NET 6 then it will fail because:

• Grpc.Net.Client doesn't set the request version in netcoreapp3.1 - https://github.com/grpc/grpc-dotnet/blob/e7fd8004c04589a9615d2859257883a8be41afde/src/Grpc.Net.Client/Internal/GrpcCall.cs#L922-L924
• The System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport switch is removed in .NET 6