ASP.NET gRPC service in IIS10 does propagate gRPC status codes

[ThisIsAlsoMyPassword]

Is there an existing issue for this?

• [x] I have searched the existing issues

Describe the bug

We have an ASP.NET application providing a number of gRPC services that works well hosted in Kestrel that we need to host with IIS10. In IIS, gRPC calls the result in a success status code work fine. The trouble begins when the service reports an error status code. Any status code in the service always results in status code 13 (Internal) in the client. After lots of searching I found some mention of similar problems, that did not seem to get to the core of the problem. Most notably an issue in the dotnet-grpc repo that lead @JamesNK to open this issue in this repo here. I can't tell if those lead to any action. It seems to have moved to .NET8 milestone but the issue seems to persist in .NET9. In the aforementioned issue the problem was complicated by the reporter using a proxy service, so I decided to create a minimal example for myself to get to the root of the problem.

Expected Behavior

An ASP.NET application hosted in IIS10 meeting the minimal requirements for servicing gRPC services propagates RpcExceptions thrown in the service correctly to the client, no matter the gRPC implementation.

Steps To Reproduce

1. Create the following minimal example, based on the Greeter-Service

GreeterService.cs

using Greet;
using Grpc.Core;

namespace Server
{
    public class GreeterService : Greeter.GreeterBase
    {
        private readonly ILogger _logger;

        public GreeterService(ILoggerFactory loggerFactory)
        {
            _logger = loggerFactory.CreateLogger<GreeterService>();
        }

        public override Task<HelloReply> SayHello(HelloRequest request, ServerCallContext context)
        {
            _logger.LogInformation($"Sending hello to {request.Name}");
            throw new RpcException(new Status(StatusCode.PermissionDenied, "error"));
        }
    }
}

Program.cs

using Grpc.Core.Interceptors;
using Grpc.Core;
using Server;
var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddGrpc(o =>
{
    o.Interceptors.Add<WriteResponseHeadersInterceptor>();
});
builder.Services.AddGrpcReflection();
var app = builder.Build();

// Configure the HTTP request pipeline.
app.MapGrpcService<GreeterService>();
app.MapGrpcReflectionService();
app.MapGet("/", () => "Communication with gRPC endpoints must be made through a gRPC client. To learn how to create a client, visit: https://go.microsoft.com/fwlink/?linkid=2086909");

app.Run();

public class WriteResponseHeadersInterceptor : Interceptor
{
    public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(TRequest request, ServerCallContext context, UnaryServerMethod<TRequest, TResponse> continuation)
    {
        // await context.WriteResponseHeadersAsync(new());
        return await base.UnaryServerHandler(request, context, continuation);
    }
}

The Program.cs contains an interceptor with the suggested workaround. The line providing the actual "fix" is commented out because the observed behavior is the same.

2. Start with IIS from Visual Studio

3. Test with grpcurl

4. Positive-Control with Kestrel

The same behavior can be observed with Postman. The clients that connect to the production server use the C++ gRPC implementation and seem to suffer the same problem.

I can't think of a more minimal example than this. As far as I can tell, all prerequisites for using gRPC in IIS10 should be met (see below) and I would expect that status codes to be propagated correctly to any client regardless of the gRPC implementation.

I'm still hoping I'm doing something wrong and there is an easy fix!

Exceptions (if any)

No response

.NET Version

9.0.103

Anything else?

Below are the version of the products I used locally for the minimal example. On the actual server where the problem surfaced we use Windows Server 2022 Version 10.0.20348 Build 20348

Version information

Visual Studio

Microsoft Visual Studio Professional 2022 (64-bit) - Current Version 17.12.5

Windows

Microsoft Windows 11 Pro 10.0.22631 Build 22631

grpcurl

v1.9.3

Build-Target

net8/net9

[JDr111]

Has there been any progress on this issue? I am encountering the same issue, and it's proving to be a significant problem in my specific use case. The issue was previously described in #47183, but let me share my own observations.

The setup

We have a C++ gRPC client connecting to an ASP.NET Core gRPC server, which can be run inside of Kestrel, but in the end we're required to deploy it to IIS in production.

The C++ client makes a call to an endpoint on the server, which is purposefully set up to produce an RpcException, that is, a header-only response in terms of HTTP/2. The call is made via HTTPS/TLS, and Wireshark was set up to decrypt the traffic.

Observed behavior - scenario 1: C# gRPC server in IIS

Here, we see the gRPC server hosted in IIS first returns a HEADERS frame with HTTP status 200 and END_HEADERS flag, which contains all response headers, including those of the RpcException (grpc-status, grpc-message):

This is followed by an empty DATA frame with the END_STREAM flag set:

The C++ client reports an error with status UNKNOWN, and message "Stream removed".

Observed behavior - scenario 2: C# gRPC server in Kestrel

The response from Kestrel is very different. The first part is a HEADERS frame with HTTP status 200, which only contains basic headers with server information, etc., but not the RpcException information:

This is followed by another HEADERS frame, which contains RpcException headers (grpc-status, grpc-message), as well as the END_STREAM flag:

Curiously, both HEADER frames have the END_HEADERS flag set. Is that supposed to happen?

No DATA frames are found in the response from Kestrel.

The C++ client reports the correct error code and message.

Observations

The big difference here is that IIS sends all response headers in a single HEADER frame, while Kestrel sends basic responses headers in one HEADER frame, and the response headers related to RpcException in another HEADER frame. I am not well versed with either the HTTP/2 or gRPC specs, so I am unsure which of the two is supposed to be correct.

This behavior is the same regardless of the version of Windows Server on which the C# gRPC server is hosted (I tried Windows Server 2022 and Windows Server 2025), and the version of .NET runtime/hosting bundle (I tried 8.0.16, 8.0.17 and 9.0.6).

One curious observation however - a C# gRPC client making a call to the same endpoint on the C# gRPC server hosted in IIS actually works, and interprets the RpcException correctly. The C# client seems smart enough to not mind the empty DATA frame IIS sends.

From these observations, I suspect the issue is either in the ANCM (ASP.NET Core Module) for IIS, or IIS itself. I do know that ASP.NET Core's support in IIS started with out-of-process hosting, which put IIS into the role of a reverse proxy, which are known to modify headers in various different ways. I don't know how in-process hosting works under the hood, but is there a possibility there's something in either the ACNM or IIS that's merging the HEADER frames?

I did notice that sometimes response headers aren't immediately sent to the client when the server is either in IIS or behind a reverse proxy. They get sent with the first DATA frame that goes from the server to the client. I first noticed this behavior when I put a gRPC server in Kestrel behind YARP reverse proxy in an unrelated project.

Similar thing occurs when the gRPC server is in IIS. I had a client-side streaming endpoint set up. The client sent the initial request, then waited for the response headers before continuing with uploading data. With Kestrel, this worked fine as the response headers arrived immediately after sending them from server-side. However, in IIS, the response headers never physically left the server machine, and I had to turn the client-side streaming endpoint into a bidirectional streaming endpoint, and send an empty message from the server to client in order to force the response headers to be physically sent to the client.

Could this be messing with the headers-only responses, that response headers aren't being sent out immediately, but rather with the first DATA frame, which is why IIS sends out an empty DATA frame when the response is header-only?

Or perhaps we just missed something when configuring IIS for gRPC, and it doesn't work "out-of-the-box"?

Why this matters

The biggest problem with this is that this issue prevents people from using gRPC's exception mechanism to implement features such as sending application-specific error codes or retry-with-backoff mechanisms. Features like these then form the base for implementing a "slots" mechanism to limit the number of clients that are allowed to call a specific endpoint on the server at the same time. This is relevant for scenarios where a large number of clients need to upload vast amounts of data at the same time. In my specific case, this can be thousands of clients with tens of gigabytes of data per client that is to be uploaded to the server (cluster) via gRPC streaming.

My use case has a strong preference for writing the client in C++, as there's a strong need to minimize external dependencies, which can be done by statically linking everything possible. Simply copy the executable to any computer, and it should run. All this while maintaining a manageable executable size (I'm at 12 MB in release configuration).

One could easily argue - just use Kestrel. Sure, but we need enterprise-grade scalability and manageability that currently only IIS can offer.

Microsoft claims IIS supports gRPC. My observation is that its support is broken.

To sum up

• C# gRPC client works with C# gRPC server hosted in IIS (and Kestrel)
• C++ and other gRPC clients do not work with C# gRPC server hosted in IIS (but work with Kestrel)

If this issue isn't resolved, I see only two options:

• Use Kestrel, which isn't ideal for large-scale enterprise deployments
• Write the client in C#, interop with C++ code for critical parts (performance and otherwise), and use Native AOT to statically link to external dependencies such as Sqlite to produce a single executable file in Release configurations. This will almost certainly result in a larger executable file, which I guess is an acceptable trade-off as long as it's not 100+ MB, however architecting a solution like this is a rather big rabbit hole I'd like to avoid if possible. And even then, who said this issue won't pop up in the C# implementation of gRPC in the future?