Content-length: 0 confuses grpc-web client

[wlhee]

A gRPC server could send back a gRPC response without any body, for example, a gRPC error response. However, some server could choose not to use "trailers-only" to send back the "grpc-status". Instead, it still uses "headers", "empty body", "trailers" HTTP/2 frames, and the grpc-status and grpc-message are set in the trailers.

There could be scenario when "content-length: 0" is added to the response header as the body is empty. For example, if the gRPC server is fronted by an envoy, the envoy could add "C-L: 0". See https://github.com/envoyproxy/envoy/issues/5554. An example response looks like this:

envoy_1   | [2021-06-19 02:40:08.061][15][debug][http] [source/common/http/conn_manager_impl.cc:1472] [C6][S17521888214259471014] encoding headers via codec (end_stream=false):
envoy_1   | ':status', '200'
envoy_1   | 'grpc-accept-encoding', 'identity'
envoy_1   | 'grpc-encoding', 'identity'
envoy_1   | 'content-type', 'application/grpc+proto'
envoy_1   | 'date', 'Sat, 19 Jun 2021 02:40:08 GMT'
envoy_1   | 'server', 'envoy'
envoy_1   | 'x-cloud-trace-context', 'a86f4f6eeee8e5baccd463176c163b05/11003727237429165764'
envoy_1   | 'content-length', '0'
envoy_1   | 'x-envoy-upstream-service-time', '170'
envoy_1   |
envoy_1   | [2021-06-19 02:40:08.062][15][debug][client] [source/common/http/codec_client.cc:130] [C7] response complete
envoy_1   | [2021-06-19 02:40:08.062][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:200] [C7] destroying stream: 0 remaining
envoy_1   | [2021-06-19 02:40:08.062][15][debug][http] [source/common/http/conn_manager_impl.cc:1488] [C6][S17521888214259471014] encoding trailers via codec:
envoy_1   | 'grpc-status', '200'
envoy_1   | 'grpc-message', 'invalid%20input'

For this case, as grpc-web filter doesn't strip content-length nor update content length after it encodes the trailers and puts it into the response body. The output from the filter could confuses the grpc-web client and makes it reject the response.

See a reproduction case in https://github.com/ridedott/grpc-web-cloud-run-issue And confirm removing content-length header makes the grpc-web client happy: https://github.com/ridedott/grpc-web-cloud-run-issue

[wlhee]

For correctness, should grpc-web filter remove C-L or update C-L to the actual size of the body?

[stanley-cheung]

@Jennnnny Can you please take a first look at this issue? This comes from an internal bug ( b/189801953 ~~will update the bug number if I find it~~ ). This could be something in the Envoy filter that's outside of this repo though.

[stefannegrea]

@stanley-cheung , I posted in https://github.com/grpc/grpc-web/issues/1050#issuecomment-888741174 before seeing this issue. I am experiencing the same problem with the javascript library. Tested with two proxies and have identical results. I would appreciate help with debugging/fixing this.

[LukeLaScala]

Hey all. What's the current status on this and are there any workarounds in the time being?

[stefannegrea]

I just retried the test case from https://github.com/grpc/grpc-web/issues/1050#issuecomment-888741174 with the latest from this repo and no change. Any guidance?

[stefannegrea]

I just found the issue and it has nothing to do with the javascript library itself.

The CORS pre-flight response has to include grpc-status,grpc-message in Access-Control-Expose-Headers otherwise the browser will ignore the two headers for on the subsequent grpc-web response. Always something to learn about CORS ...

In my test cases, with both envoy and grpc-wrapper, I was making the mistake of allowing the grpc-status,grpc-message on the actual request without flagging the two headers as allowed via CORS pre-fight response. This got me really confused, I was able to see the two error headers in the response in the browser but the error parsed by the library would always be status code 2 with incomplete response. The library is working correctly since it does not have the error message information without the headers being pre-allowed. As soon as I made this small change all the errors were returned consistently. From my testing, content-length or other headers do not have any impact on correctly parsing the error status once the headers are allowed in the CORS pre-flight exchange.

@wlhee, can you please try the sample code you have with this small update? If you add grpc-status,grpc-message at the end of https://github.com/ridedott/grpc-web-cloud-run-issue/blob/master/envoy/envoy.template.yaml#L65 (cors.allow_headers) , your test application should be able to parse the error information consistently.

[efossvold]

#1050 (comment) solved it for us.

[RmStorm]

I think that this issue can be closed since the actual problems is indeed the response headers and not anything in grpc-web.. I also ran into this issue and Stefan's solution worked for me.

[sampajano]

@RmStorm Thanks for the comment!

I'm closing this bug now based on the comment from @RmStorm and @stefannegrea above (thanks both!!). Please feel free to re-open if the issue remains.

Thanks!