Proxying grpc/h2c requests fail with "server closed the stream without sending trailers"

[ahmetb]

I am trying to build a small man-in-the-middle proxy tool for inspecting gRPC client/server traffic. So I am using this goproxy to almost build a reverse-proxy server.

What I'm seeing is that goproxy actually proxies fine, and gets the response bytes back from a gRPC server. However the client sees a code = Internal desc = server closed the stream without sending trailers from grpc-go.

The same implementation works fine when httputil.ReverseProxy is used.

---

Repro steps

1. For this example I'll be using grpc-go route_guide client and server sample: https://github.com/grpc/grpc-go/blob/master/examples/route_guide/server/server.go

2. I deploy the server somewhere that it runs with HTTPS:// (e.g. Google Cloud Run). So my server URL is actually https://routeguide-2wvlk7vg3a-uc.a.run.app:443

3. Now I write a small program using goproxy. All it does is to log req/resp and modify the "Host" and "req.URL" to the value in the previous step.

4. Add a /etc/hosts entry routeguide 127.0.0.1

5. Run the program in step 3 go run . on port 8080

6. Run the route_guide/client program, but point it to the local proxy: go run ./client -server_addr=routeguide:8080

Observed result:

route_guide/client program fails with this error:

go run ./client -server_addr=routeguide:8080
2020/10/09 11:42:47 Getting feature for point (409146138, -746188906)
2020/10/09 11:42:49 &{0xc0001cea80}.GetFeatures(_) = _, rpc error: code = Internal desc = server closed the stream without sending trailers:
exit status 1

meanwhile the proxy shows everything is actually fine and proxy has done its job (req returns HTTP 200 + resp body is there):

2020/10/09 11:42:47 [001] INFO: Got request /routeguide.RouteGuide/GetFeature routeguide:8080 POST http:///routeguide.RouteGuide/GetFeature
2020/10/09 11:42:47 [proxy] start: POST url=http:///routeguide.RouteGuide/GetFeature hdrs=3 trailers=0
2020/10/09 11:42:47 	> HDR Te=[]string{"trailers"}
2020/10/09 11:42:47 	> HDR Content-Type=[]string{"application/grpc"}
2020/10/09 11:42:47 	> HDR User-Agent=[]string{"grpc-go/1.33.0-dev"}
2020/10/09 11:42:47 [director] host=routeguide:8080 url=http:///routeguide.RouteGuide/GetFeature
2020/10/09 11:42:47 [director] rewrote host=routeguide:8080 to="https://routeguide-2wvlk7vg3a-uc.a.run.app/routeguide.RouteGuide/GetFeature"
2020/10/09 11:42:47 [001] INFO: Sending request POST https://routeguide-2wvlk7vg3a-uc.a.run.app/routeguide.RouteGuide/GetFeature
2020/10/09 11:42:49 [001] INFO: Received response 200 OK
2020/10/09 11:42:49 [proxy]   resp: code=200 hdrs=4 trailers=0
2020/10/09 11:42:49 	< HDR Content-Type=[]string{"application/grpc"}
2020/10/09 11:42:49 	< HDR Date=[]string{"Fri, 09 Oct 2020 18:42:49 GMT"}
2020/10/09 11:42:49 	< HDR Server=[]string{"Google Frontend"}
2020/10/09 11:42:49 	< HDR Alt-Svc=[]string{"h3-Q050=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-27=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""}
2020/10/09 11:42:49 [001] INFO: Copying response to client 200 OK [200]
2020/10/09 11:42:49 [001] INFO: Copied 84 bytes to client error=<nil>

Alternative implementation with httputil.ReverseProxy

As an alternative to goproxy, I reimplemented the same using httptuil.ReverseProxy and it seems to be working just fine. https://gist.github.com/ahmetb/cf18930fc5a863253c5f7f59ca646ddb

---

Conclusion: I think there's something in goproxy codebase that's not compatible with h2 or grpc. It seems possibly the trailers are not flushed out –I am not sure.

At any rate, if we can fix this, a gprc proxy can be built on top of this tool nicely.