Error information is not correctly sent to client when hosted using HTTP.sys

[ArthurHNL]

What version of gRPC and what language are you using?

C#, see the following excerpt of the .csproj file for the gRPC version:

  <ItemGroup>
    <PackageReference Include="Grpc.AspNetCore" Version="2.53.0" />
    <PackageReference Include="Grpc.AspNetCore.Server.Reflection" Version="2.53.0" />
  </ItemGroup>

What operating system (Linux, Windows,...) and version?

For compilation: Windows 10 Version 22H2 (Build 19045.2846) For execution: Windows Server 2022 Version 21H2 (Build 20348.1668)

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

dotnet --info on the PC used for compilation

>dotnet --info
.NET SDK:
 Version:   7.0.203
 Commit:    5b005c19f5

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19045
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.203\

Host:
  Version:      7.0.5
  Architecture: x64
  Commit:       8042d61b17

.NET SDKs installed:
  7.0.203 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

dotnet --info on the server running the app

>dotnet --info

Host:
  Version:      7.0.5
  Architecture: x64
  Commit:       8042d61b17

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

What did you do?

I host a gRPC service using HTTP.sys and call an RPC raises an RpcException with status code 9 (FailedPrecondition) and the error message test error.

What did you expect to see?

When an RpcException is thrown, I expect the error information to be conveyed to the client as shown in the gRPC UI client below, this example is using Kestrel which exhibits the correct behavior:

What did you see instead?

The error details are not sent to the client. Instead, a message is displayed stating that 'the server closed the stream without sending trailers':

However, the log of the server claims that an error response is sent:

dbug: Grpc.AspNetCore.Server.ServerCallHandler[10]
      Reading message.
trce: Grpc.AspNetCore.Server.ServerCallHandler[12]
      Deserializing 0 byte message to 'GrpcTestHttpSys.HelloRequest'.
trce: Grpc.AspNetCore.Server.ServerCallHandler[13]
      Received message.
info: Grpc.AspNetCore.Server.ServerCallHandler[7]
      Error status code 'FailedPrecondition' with detail 'test error' raised.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /greet.Greeter/Error'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST https://<HOSTNAME>:5930/greet.Greeter/Error application/grpc - - 200 0 application/grpc 31.4102ms

Anything else we should know about your project / environment?

• Regular RPC calls that do not raise errors do not appear to be affected. I have not tested streaming calls.
• I have attached a demo solution that hosts the same service that always throws an RpcException and that is hosted using both Http.sys and Kestrel.
  • GrpcTestHttpSys.zip
  • This zip also contains the detailed logs of both the gPRC UI client and the server in both scenarios (HTTP.sys and Kestrel).
• We also see this incorrect behavior on IIS and it is also important for us that this is fixed in IIS. However we suspect that both are signs of the same problem caused by Http.SYS as IIS uses HTTP.sys under the hood.
• We really want to migrate to gRPC but this is holding us back. We cannot migrate our WCF services to gRPC without working port sharing and on Windows Server HTTP.sys or IIS is the easiest way to achieve this.

[ArthurHNL]

@JamesNK Do you have any idea what is going on here?

[ArthurHNL]

@mgravell Since you assigned yourself to this issue, can you provide an update? If you need anything (eg more info) please let me know because this is blocking quite some internal things for us.