ESPv2 authentication error getting mapped to StatusCode.UNKNOWN

[j4ckofalltrades]

I have a gRPC endpoint that sits behind ESPv2 with JWT authentication configured and I'm trying to configure a retry policy that will retry requests when say an invalid / expired JWT is passed in the request.

I was expecting that a HTTP 401 response from ESPv2 would be mapped to StatusCode.UNAUTHENTICATED but instead it gets mapped to StatusCode.UNKNOWN.

Sample server response:

< HTTP/2 401
< date: Tue, 19 Jan 2021 06:54:03 GMT
< content-type: application/json
< content-length: 40
< strict-transport-security: max-age=15724800; includeSubDomains
< access-control-allow-origin: *
< access-control-allow-credentials: true
< access-control-allow-methods: PUT, HEAD, GET, POST, OPTIONS
< access-control-allow-headers: Authorization, X-User-Agent, X-Grpc-Web, Content-Type
<
{"code":401,"message":"Jwt is missing"}

This then gets mapped to:

{
  code: StatusCode.UNKNOWN,
  message: 'Unknown Content-type received.',
  metadata: {},
}

Relevant code seems to be here: https://github.com/grpc/grpc-web/blob/master/javascript/net/grpc/web/grpcwebclientreadablestream.js#L154-L159

Is there a way to do this?

[j4ckofalltrades]

Turns out I was missing Access-Control-Expose-Headers: Grpc-Message, Grpc-Status in my CORS policy; closing.

[LiBa001]

I got the exact same error, but adding those headers doesn't change anything. I think the problem is, that the server returns a content type application/json in case of an error, when usually it returns application/grpc-web+proto. Therefore, it is an unknown content type. Now I don't know whether the problem lies somewhere in my ESPv2 configuration which causes it to return the wrong content type or it is normal for applictation/json to be returned on errors (probably the former).

@j4ckofalltrades did you possibly change anything else, besides the CORS expose headers? Cause I don't see how they would've changed the content type.

[j4ckofalltrades]

@LiBa001 Thanks for pointing this out.

@j4ckofalltrades did you possibly change anything else, besides the CORS expose headers? Cause I don't see how they would've changed the content type.

Haven't found a solution to this one unfortunately (the other issue I had with the CORS config was unrelated to this one and I forgot to update this issue) -- reopening this issue.

[nsadeh]

Hey, my hunch is that the network call is correctly responding with a 401 and that your gRPC client maps that to an UNKNOWN since it cannot parse the headers returned by ESPv2. Basically ESP tried to authenticate your request, couldn't, so it responded to your client using an application/json content type saying it couldn't authenticate. Your gRPC client couldn't understand what it was saying so it just said "content type unknown". ESP didn't so much "change" the content type as didn't really care about your original content type as it got stuck in the auth stage.

I am conjecturing this based on my current experience, I am trying to get auth setup on a gRPC backend as well with ESPv2 proxy, and I see my network results as

{"message":"UNAUTHENTICATED:Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.","code":401}

but my RPC client says Unknown Content-type received. In this case, it means that the client received an unknown content type in the form of JSON.

[N0N4M3pl]

Hi. I have the same problem. In my scenerio i make grpc-web request with not valid / expired access token (JWT):

Request URL: https://devbackendapi.abc.com/pl.abc.api.pub.grpc.position.v1.PositionService/SubscribePositions
Request Method: POST
accept: application/grpc-web-text
accept-encoding: gzip, deflate, br
authorization: Bearer expired.access.token
content-length: 8
content-type: application/grpc-web-text
origin: https://localhost.abc.com
referer: https://localhost.abc.com/
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36
x-grpc-web: 1
x-user-agent: grpc-web-javascript/0.1

It goes through: webbrowser --> nginx (as proxy) --> Istio (1.12) --> ... In Istio I have added RequestAuthentication (Istio Gateway Envoy configuration) which it checks token. If it is correct and valid it pass request to PositionService. If it is not correct (like in this example) it response:

Status Code: 401
ontent-length: 29
content-type: text/plain
date: Wed, 18 May 2022 16:45:55 GMT
server: nginx
www-authenticate: Bearer realm="https://devbackendapi.abc.com/pl.abc.api.pub.grpc.position.v1.PositionService/SubscribePositions", error="invalid_token"

grpc-web library check response - look at this: https://github.com/grpc/grpc-web/blob/35284bfe156fc41bbcdd554ac423a587d93ff8da/javascript/net/grpc/web/grpcwebclientreadablestream.js#L138-L159 First it checks Content-Type and if it is text/plain it returns new RpcError(StatusCode.UNKNOWN, 'Unknown Content-type received.'));

In my opinion server/backend can response with that simple 401 text/plain message. Its correct and recommended to not tell too much about reason. Beacuse token was incorrect it didnt pass to PositionService and couldnt be in type of grpc error. I think grpc-web library should operate on this type of errors.

If im wrong please write how to resolve this problem. How to get 401 status code from this error.

[ericb-summit]

espv2 is envoy based. Assuming it is possible to control the config, you could use a local reply to rewrite the 401 to something that grpc-web will consume. This works.

              local_reply_config:
                mappers:
                - filter:
                    status_code_filter:
                      comparison:
                        op: EQ
                        value:
                          default_value: 401
                          runtime_key: "%RESPONSE_CODE%"
                  headers_to_add:
                    - header:
                        key: "Content-Type"
                        value: 'application/grpc-web+proto'
                      append: false
                    - header:
                        key: "grpc-message"
                        value: 'Not authenticated'
                      append: false
                    - header:
                        key: "grpc-status"
                        value: '16'
                      append: false
                  status_code: 200
                  body:
                    inline_string: ""
                  body_format_override:
                    content_type: "application/grpc-web+proto"
                    text_format: ""

[j4ckofalltrades]

@ericb-summit Thank you, this works for my use case. Marking issue as closed.