net/http: Unexpected "define Request.GetBody to avoid this error" error after setting Request.GetBody
Go version
go version go1.23.0 darwin/arm64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/xxx/Library/Caches/go-build'
GOENV='/Users/xxx/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
What did you do?
I have not been able to make a stand alone example, but will walk through the exact code path that is triggered.
The offending code is as follows:
go func(client *http.Client){
dataToPost := []byte("somedata")
// if we don't set the body here, the first attempt seems send with an empty body even though GetBody is set
req, err := http.NewRequest(http.MethodPost, "http://127.0.0.1/someapi", bytes.NewBuffer(dataToPost))
if err != nil {
log.Fatalln("failed to create request!")
}
req.GetBody = func() (io.ReadCloser, error) {
return io.NopCloser(bytes.NewBuffer(dataToPost)), nil
}
res, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
}(someSharedClient)
Which under very particular circumstances, the call to client.Do returns the error http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error, even though GetBody has been set.
To get here, the request must be redirected followed by a retry-able HTTP/2 error such as GOAWAY. (I believe it also may require a cached HTTP/2 connection from a previous call).
How I managed this was the following setup:
Load generating go program (the one getting the error) with at least two go routines sending on the same client <-> nginx minikube ingress <-> some kubernetes service
The code path is as follows:
- The first attempt to send the request gets redirected to https (with support for HTTP/2), entering the second iteration of this loop. I did not determine if a redirect from an HTTP/2 endpoint to another HTTP/2 endpoint would cause the same errors.
- Since the first loop added the request to the reqs slice here we enter this if statement.
- Notice the
GetBodyfunction does not get set when the original request is copied here. - The new request enters the http2 round tripper here.
- The second attempt to send the request is met by a retry-able error here (in my case it appears to be a GOAWAY sent by nginx after some number of requests as described), and
http2shouldRetryRequestis called. - Since
GetBodywas not copied on the redirect, this section fails to copy the body, and we reach the resulting error here.
What did you see happen?
I received an error telling me to set GetBody to avoid said error, even though GetBody had in fact been set.
What did you expect to see?
I would expect either the GetBody function pointer to be copied on redirect, or the error returned to better reflect the actual error condition.
Related Issues and Documentation
- net/http, x/net/http2: client loses the request body when retrying after HTTP2 server close connection gracefully(GOAWAY) #39086 (closed)
- net/http: Re-connect with upgraded HTTP2 connection fails to send Request.body #32441 (closed)
- net/http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error #62453 (closed)
- x/net/http2: investigate data race in Transport Request.Body reuse when GetBody is not defined #25009 (closed)
- net/http, x/net/http2: http.readTrackingBody data race after received GOAWAY frame #41234 (closed)
- net/http: request not rewound in certain scenario if HTTP/2 stack is used #49609
- net/http: clarify whether Requests can be reused in HTTP/2 #45311
- net/http: http2 close request body and read it again #59426 (closed)
- net/http, x/net/http2: support http2requestBody.Close() being called multiple times concurrently #51197 (closed)
- x/net/http2: PROTOCOL_ERROR after GOAWAY #32112 (closed)
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
CC @neild, @rsc.
Possibly related flakes #66521, #66477. Some of the issues (#62453, #25009) pointed to by gaby mention the same error message.
Related Caddy Issue: Transport received Server's graceful shutdown GOAWAY #3428