See HTTP/2 error code 'PROTOCOL_ERROR`, after enabling `HttpKeepAlivePingPolicy.Always`

[nlyu]

Describe the bug

We are a gateway on NET8, YARP 2.2

Caller --> Gateway --> Client service

Recently we changed our socket http client handler (it owns the gateway --> client service part proxy) setting from

handler.KeepAlivePingPolicy = HttpKeepAlivePingPolicy.WithActiveRequests

to

handler.KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always;
handler.KeepAlivePingDelay = TimeSpan.FromSeconds(30);
handler.KeepAlivePingTimeout = TimeSpan.FromSeconds(10);

trying to see if this can help us enhance some HTTP2 wired cancellation issue.

But we see after this change, many request which are previously getting timeout are now reporting

"IslandGateway.Core.Service.ErrorHandling.ProxyException: YARP error YarpRequest\r\nSystem.Net.Http.HttpRequestException: 

The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)\r\nSystem.Net.Http.HttpProtocolException: 

The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)"

o_O I think there is some change in HTTP2 behavior but not sure why. Any good suggestions on tweeting this handler.KeepAlivePingPolicy? Not sure if this issue should be posted in YARP here or DOTNET

To Reproduce

Further technical details

• Include the version of the packages you are using
• The platform (Linux/macOS/Windows)

[MihaZupan]

Do you know what server is running behind the proxy? We've seen some cases of servers/firewalls being very aggressive about not accepting too many pings, especially when not processing requests. E.g. https://github.com/dotnet/runtime/issues/62216

Is this happening regularly / something you can reproduce in isolation?

cc: @antonfirsov in case this sounds familiar

[antonfirsov]

KeepAlive pings are not included in the RTT ping accounting logic which can theoretically lead to unsolicited pings even with a well-behaving server, although there were no previous reports of such an issue.

what server is running behind the proxy? something you can reproduce in isolation?

These are key questions. Also, what does "client service" do and what kind of traffic is going through YARP? It would be also great to collect client and server side traces if possible.

[nlyu]

The client team reporting this issue are on: NET7 + Kestrel or NET Framework + Owin

ok i will work with one team to enable debug log. Yea the most interesting thing is all the timeout/deliberately cancelled request transformed to this HTTP2 protocol error after our change.

[MihaZupan]

I'll close this as not actionable at the moment. Let us know if you're able to get more info.