grpc-status is not returned when making unary call

[chengyaxue]

trafficstars

I've build a grpc server and exposed the server by envoy-proxy. I found that when grpc server throws GrpcStatusException, grpc-web always gets error saying incomplete response if making unary calls. I was able to get grpc-status on 'status' event if I invoke stream APIs. Please advise how to debug this issue. Thanks!

Version

• envoyproxy 1.17.0
• google-protobuf": "^3.15.3"
• grpc-web": "^1.2.1"

I generated protobut file by running protoc -I. proto/myservice.proto \ --js_out=import_style=commonjs:src --grpc-web_out=import_style=commonjs,mode=grpcweb:src

JS Code

var myservice = require("./my_service_pb");
var myservice_client = require("./my_service_grpc_web_pb");
var client = new myservice_client.Foo(
  "http://localhost:28080"
);
var request = new myservice.BarRequest();
client.bar(request, {}, (err, response) => {
   console.log(error, error)
}

output

Content.js:33 error {code: 2, message: "Incomplete response"}

envoy.yaml

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: main-listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: grpc-backend-services
                  max_grpc_timeout: 0s
              cors:
               allow_origin:
               - "*"
               allow_methods: GET, PUT, DELETE, POST, OPTIONS
               allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
               max_age: "1728000"
               expose_headers: custom-header-1,grpc-status,grpc-message
               enabled: true
          http_filters:
          - name: envoy.grpc_web
          - name: envoy.cors
          - name: envoy.router
  clusters:
  - name: grpc-backend-services
    connect_timeout: 0.25s
    type: logical_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: localhost
        port_value: 6080

[RohitRox]

Might be related https://github.com/grpc/grpc-web/issues/951 and https://github.com/grpc/grpc-web/pull/903

[stefannegrea]

I am having the exact same issue and did some testing. All gPRC servers (regardless of implementation language) should either return a response without an error or an error without a response. Both envoy and improbable-eng proxy send the gRPC errors to the browser via headers with an empty payload and 200 response code. The current js client implementation makes a mistake and assumes an empty response is an incomplete response.

In my testing, I was attempting to return unautheticated errors from a golang gRPC server.

Here is some sample js code:

      try {
        const request = new MyRequest();
        await myservice.myendpoint(new MyRequest(), null);
      } catch (err) {
        console.log("code:", err.code);
        console.log("message:", err.message);
      }

This is what I get back in the browser:

General
 Request Method: POST
 Status Code: 200 

Headers
  ....
  grpc-message: unauthenticated
  grpc-status: 16
   ...

So when the gRPC server errors out, the proxy returns 200, empty body, and two special headers. The js code above prints code:2 message: Incomplete response when I return any type of error from the gRPC server. In my case I am returning unauthenticated from the gRPC server. This behaviour is consistent with both envoy and improbably-eng proxy.

The generated js/ts code looks fine, however the library fails to extract the error codes from the headers and always assumes an empty payload is an incomplete payload. So my guess there is something that got introduced recently in the javascript library but I am yet to find it.

I would appreciate some help fixing or debugging this.

[LukeLaScala]

I was able to fix this issue actually. The issue I was having was that the grpc headers in the response from envoy were being filtered out and thus the grpc web client was unable to parse the response correctly.

[stefannegrea]

@LukeLaScala, what headers were filtered out? Which headers are important to correctly parsing a failed response?

[stefannegrea]

@LukeLaScala, thanks a lot for the hint, I was able to pin-point the issue; please see https://github.com/grpc/grpc-web/issues/1101#issuecomment-939303449 for all the details.

@chengyaxue, do you mind testing again with the changes suggested in https://github.com/grpc/grpc-web/issues/1101#issuecomment-939303449 ? Just need to add grpc-status,grpc-message at the end of allow_headers: line in envoy.yaml config file.

[chengyaxue]

@LukeLaScala, thanks a lot for the hint, I was able to pin-point the issue; please see #1101 (comment) for all the details.

@chengyaxue, do you mind testing again with the changes suggested in #1101 (comment) ? Just need to add grpc-status,grpc-message at the end of allow_headers: line in envoy.yaml config file.

Hi @stefannegrea, we've recently found out the root cause. It's because our server added header content-length: 0 when there is no content. Since the PLB(Nginx) is not expecting any trailer headers to be added when response length is known to be 0, grpc-web client couldn't get the status.

[efossvold]

We had a similar issue where gRPC error responses always would result in "incomplete response" on the client. Envoy would remove the "grpc-status" and "grpc-message" headers even though they were added to"expose_headers" in the envoy.yaml config. We tried all solutions mentioned here and in #1101 and #951.

What finally solved it was having envoy to remove the content-length header as per the comment in this issue (see the PR).

[Omisamuel]

Thank you for sharing this hint.