Retryability on an incoming GOAWAY frame

[ruccho]

Some implementations of HTTP/2-compliant servers may set an upper limit on the cumulative number of requests per connection. When a request exceeding this limit is sent, a GOAWAY frame is sent from the server. The GOAWAY frame containsLast-Stream-ID, and streams with IDs lower than the last stream id will continue until completion, but streams with IDs higher than the last stream id will be canceled if the client has already sent one. Canceled requests appear as an exception in YetAnotherHttpHandler.

IOException: client error (SendRequest): http2 error

According to RFC 9113, these canceled streams can be safely retried on a new connection. However, according to hyperium/hyper#2500, hyper has already dropped the information necessary for retries when it receives GOAWAY and cannot automatically retry them. It says that it must be retried at a higher layer.

Possible Solutions

• Retry at the YetAnotherHttpHandler layer.
  • It may be difficult to hold the information necessary to retry until the completion of the request.
• Retry at the user code layer.
  • Currently, the exception doesn't tell us whether the request can be safely retried.

Reproduction

Unfortunately, I was not able to find a way to reproduce this behavior in ASP.NET Core, but was able to reproduce it with NGINX instead. Enable HTTP/2 in nginx.conf and set the keepalive_requests value to a smaller value such as 10.

If you send more requests than keepalive_requests in a row with YetAnotherHttpHandler, an exception will be thrown.

    [Fact]
    public async Task GracefulShutdownRetryability()
    {
        using var httpClient = new HttpClient(new YetAnotherHttpHandler()
        {
            SkipCertificateVerification = true
        }, true);

        // Warms up the connection
        // Before the first connection is established and pooled, multiple requests may create multiple connections.
        {
            var request = new HttpRequestMessage(HttpMethod.Get, $"https://localhost/");
            await httpClient.SendAsync(request);
        }

        List<Task> tasks = new();
        for (var i = 0; i < 20; i++) // send more than keepalive_requests
        {
            var request = new HttpRequestMessage(HttpMethod.Get, $"https://localhost/");
            tasks.Add(httpClient.SendAsync(request));
        }

        var ex = await Record.ExceptionAsync(async () => await Task.WhenAll(tasks));

        testOutputHelper.WriteLine(ex?.ToString() ?? "no error");

        Assert.IsType<HttpRequestException>(ex);
    }

[mayuki]

We can now reproduce this in our environment and identify request cancellations with the GOAWAY frame. However, we need to decide whether to handle retries on the Rust side or propagate errors so they can be handled in user code.

Handling retries on the Rust side may require a reconsideration of the code flow.

On the other hand, handling retries in user code is straightforward, but we need to consider a mechanism for propagating the error. This approach also feels slightly unnatural compared to the behavior of SocketsHttpHandler.