Decoding grpc-status-details-bin ?

[jscti]

Hi there

Server side, we send an error withDetails (in GO) :

st2 := status.New(codes.Aborted, "failed to map")
    st2.WithDetails(&Error{
        Code:     1,
        Message: "error",
    })

Client side (js / typescript), we need to retrieve these details somehow but all we receive is a serialized string (trailers.get('grpc-status-details-bin') ) .. how can we decode it ?

Thanks

[johanbrandhorst]

Go encodes the metadata as base64 after marshalling the proto message, so you should be able to base64 decode followed by a unmarshalling of the data into a google.rpc.Status type.

[johanbrandhorst]

See https://github.com/grpc/grpc-go/blob/ca62c6b92c334f52c7e48b7cbf624f3b877fb092/internal/transport/http_util.go#L317 for how its done in Go clients. Should be very applicable to JS clients too. We may need to add some convenience function for this, but for now this is what you will need.

[jscti]

Thanks for the (very) quick reply.

DIdn't recognize a base64 string, but yeah, decoded it this way and it's OK ;)

Thanks

[jscti]

Struggling for a few hours now, I can't figure out how to parse the base64decoded string. Nothing seems related on node_modules/grpc or node_modules/grpc_web_client. Can't find either any grpc/status look-alike file.

If you have any clue ^^

Server side :

st2 := status.New(st, "test")
    st2, _ = st2.WithDetails(&gateway_public.Error{
        ErrorType: gateway_public.ErrorType_SPONSORSHIPCODE_CAPTCHA_VERIFICATION_FAILED,
        Message:   "A",
    }, &gateway_public.Error{
        ErrorType: gateway_public.ErrorType_SPONSORSHIPCODE_CAPTCHA_VERIFICATION_FAILED,
        Message:   "B",
    }, &gateway_public.Error{
        ErrorType: gateway_public.ErrorType_SPONSORSHIPCODE_CAPTCHA_VERIFICATION_FAILED,
        Message:   "C",
    })

Client side, I receive :

test1
(type.googleapis.com/gateway.public.ErrorA1
(type.googleapis.com/gateway.public.ErrorB1
(type.googleapis.com/gateway.public.ErrorC

[johanbrandhorst]

Yeah that looks right, but you still need to parse the any.Any details in your status.Status type. You need to switch over the typeURL (e.g. type.googleapis.com/gateway.public.Error) and unmarshal the value into the correct type.

[jscti]

You totally lost me there grpc-web-client doesn't have a Status class/object to load the entire response in one go. Do I have to parse the response as a string ? Split over line-break, remove last numeric char if it exists ? (wtf?).

[johanbrandhorst]

OK I thought you had parsed this into a status type already, but I assume this string is just what you go from base64 decoding the trailer then? Here's what you need to do:

1. You need to generate a JS type for the google.rpc.Status type (https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto). I don't know if this exists pre-generated anywhere.
2. Importing from the previously generated file, you need to create a Status instance by marshalling from the decoded base64 string. I can't remember the exact method names, but it should be obvious.
3. Now that you have a Status type, you will see that it has 3 fields: Message, Code and Details. Details is an array of any.Any types.
4. For each element in the Details field of your status message, you will need to find the correct pre-generated type based on the typeURL field (for example, type.googleapis.com/gateway.public.Error means you should use the pre-generated type for this error).
5. Once you've found the type, you again need to marshal this message from the value field in the any.Any element in the details array.

That will give you your 3 error details marshalled into JS types.

[jscti]

Thanks for the help !

Stuck at step 2 :D

Tried :

const str = atob(metadataa.get('grpc-status-details-bin')[0]);
const bytes = new Uint8Array(Math.ceil(str.length / 2));
for (let i = 0; i < bytes.length; i++) {
    bytes[i] = str.charCodeAt(i);
}
const status = Status.deserializeBinary(bytes);

But decoder triggers an error ... "Failure: Decoder hit an error" Can't find a proper string to base64 helper ..

[johanbrandhorst]

You still need to base64 decode it first. What does atob do?

[johanbrandhorst]

You should be able to get a Uint8Array from a base64 decoder of the original string.

[jscti]

atob returns a string :/

[johanbrandhorst]

Does creating a Uint8Array from the string work? Anyway, that issue is clearly outside the scope of this issue. I think we can definitely do some work to make this easier for users, but I've given the basic instructions.

[jscti]

So, I got it working with a (not-so-pretty) double deserialization (grpc-go can only send any[] as details and not ErrorStatus[]) :

onEnd: (code: grpc.Code, message, responseMetadata: grpc.Metadata) => {
    if (responseMetadata.has('grpc-status-details-bin')) {
        try {
            const errDetails = atob(responseMetadata.get('grpc-status-details-bin')[0]);

            details = ErrorStatus.deserializeBinary(stringToUint8Array(errDetails))
                .getDetailsList()
                .map(detail => ErrorStatusDetail.deserializeBinary(detail.getValue_asU8()));
        } catch (error) {}
    }
[...]
}

function stringToUint8Array(str): Uint8Array {
    const buf = new ArrayBuffer(str.length);
    const bufView = new Uint8Array(buf);
    for (let i = 0; i < str.length; i++) {
        bufView[i] = str.charCodeAt(i);
    }
    return bufView;
}

and proto :

message ErrorStatusDetail {
  ErrorStatusDetailType type = 1;
  string message = 2;
}

message ErrorStatus {
  int32 code = 1;
  string message = 2;
  repeated google.protobuf.Any details = 3;
}

Thanks for your help

[johanbrandhorst]

Nice work! Just a little clarification - the double marshalling is a consequence of using any.Any - not a limitation in grpc-go.

[jscti]

Yep but when using directly ErrorStatusDetail in the proto + single marshalling, I get a "Assertion failed" error

[nhooyr]

This is really confusing. A small helper would go long way in clarifying how this works.

[jesushernandez]

Hello @jscti, what client implementation were you using here? Improbable's or this repository's? We currently use improbable's in most of our clients and were going to give a go at the 'official' one (we've generated our client with typescript support).

We use grpc-go in our services and Improbable's excellent proxy implementation (we currently wrap a grpc-go server using their utils) which allows us to embed some internal authentication logic.

We can successfully make requests and the proxy seems to be completely compatible with this new client.

However, we rely a lot on adding custom error payloads via status.WithDetails, which the grpc-go server implementation writes as the header grpc-status-details-bin, as described in this thread.

The problem is that at the moment there is no way to access this header from the API exposed by these clients. We would expect it to be included here, but the client does not even check for the header.

What are your thoughts about including this header data as part of the error callback object? We would happily add it and submit a patch.

It's not a huge issue for us at the moment, as we still have improbable's clients. But it'd be great to see this client as feature-rich as Improbable's, which allows you to do this.

Thanks a lot!

[NobodyXiao]

Hi @jscti , I have similar need with you. We write grpc server in golang and client in angular. We also read the article called "Advanced gRPC Error Usage" written by @johanbrandhorst . Now my question is that I can't get metadata, and our error return only have two fileds, not including response metadata. The code as below. is some code wrong?

server side: func (s *Server) SayHello(ctx context.Context, in *HelloRequest) (*HelloReply, error) { log.Println("test") st := status.New(codes.InvalidArgument, "invalid username") desc := "The username must only contain alphanumeric characters" v := &errdetails.BadRequest_FieldViolation{ Field:"username", Description: desc, } br := &errdetails.BadRequest{} br.FieldViolations = append(br.FieldViolations, v) st, err := st.WithDetails(br) if err != nil { // If this errored, it will always error // here, so better panic so we can figure // out why than have this silently passing. panic(fmt.Sprintf("Unexpected error attaching metadata: %v", err)) } return nil, st.Err()

// return &HelloReply{Message: "OFF SayHello(2.4) " + in.Name}, nil

}

client side: sayHello() { let serverSerice = new serviceClient.GreeterClient('http://abc.super.com:8080',null,null); let request = new HelloRequest(); let call = serverSerice.sayHello(request, {}, function(err:grpcWeb.Error, response:HelloReply) { console.log('response',err,response); //err >> {code: 3, message: "invalid username"}, response >> null, we can not get metadata }); }

proto file:

syntax = "proto3";

package helloworld;

service Greeter { // unary call rpc SayHello (HelloRequest) returns (HelloReply); // unary call - response after a length delay rpc SayHelloAfterDelay (HelloRequest) returns (HelloReply); }

message HelloRequest { string name = 1; }

message HelloReply { string message = 1; }

What can we do to solve this problem? Looking for your reply. Thank you !

[johanbrandhorst]

@NobodyXiao, it's worth inspecting the message sent from the server over the wire (in your browser network tab) to see if the error is in the client or the server.

[Siceberg]

I have the same problem as @NobodyXiao , is there a specific solution now?

[at-ishikawa]

I have the same problem. I want to do the same thing as @NobodyXiao , and it's really helpful if what @jesushernandez is mentioned is implemented in these clients. Because google has design guide to use error details for handling errors, which is written here, I think error details is better to be included in grpc.Web.Error.

[RXminuS]

We have the same problem, expected to have grpc-status-details-bin decoded into Status details. Should be a relatively easy fix though as suggested by @jesushernandez ; just checking if the header exists and deserializing the blob into the correct object.

I might have time for a PR next week if @jesushernandez hasn't started yet? Otherwise, I might put up a git bounty instead if anyone wants to join?

[tooolbox]

Yeah @RXminuS I just crashed into this. Did you start on a PR? Or @jesushernandez ?

I would do it myself even but it's vaguely daunting. I guess we'd need something like:

const GRPC_STATUS_DETAILS = "grpc-status-details-bin";
// ...
    var responseHeaders = self.xhr_.getResponseHeaders();
    if (GRPC_STATUS in responseHeaders &&
        Number(self.xhr_.getResponseHeader(GRPC_STATUS)) != StatusCode.OK) {
      self.onErrorCallback_({
        code: Number(self.xhr_.getResponseHeader(GRPC_STATUS)),
        message: self.xhr_.getResponseHeader(GRPC_STATUS_MESSAGE),
        details: self.xhr_.getResponseHeader(GRPC_STATUS_DETAILS)
      });
    }

So that's simple enough, no? But then let's say I'm attaching DebugInfo from the errdetails package as described by Mr. @johanbrandhorst , well then I confess I wouldn't know what to do next, or how this sort of thing fits into the overall project.

EDIT: Little more data in grpc/grpc-node#184 and the node-grpc-error-details package, perhaps?

[them0ntem]

I also have the same problem. I can't get metadata on status callback and error have only two fields for status code and message.

I followed the code to identify the reason for discrepancy and if I understand correctly. At line 129 and line 136 of grpcwebclientreadablestream.js byte array is created, which is used later to parse http1 headers and value of bytearray is either set from response or responseText field from XHR and both would be blank in case of error.

https://github.com/grpc/grpc-web/blob/c6af7c61301cebf20b7b27108401117c70e063e1/javascript/net/grpc/web/grpcwebclientreadablestream.js#L125-L135

Causing block which is meant to parse http1 headers, and call status callback/promise to skip. https://github.com/grpc/grpc-web/blob/c6af7c61301cebf20b7b27108401117c70e063e1/javascript/net/grpc/web/grpcwebclientreadablestream.js#L141-L181

Reference code used to Test and Debug

Server Side:

func (s Server) Backspin(ctx context.Context, req *proto.Ping) (*proto.Pong, error) {
	if req.GetMsg() == "ping" {
		return &proto.Pong{
			Msg: "pong",
		}, nil
	}

	errorStatus := status.New(codes.InvalidArgument, "wrong serve")
	errorStatus, err := errorStatus.WithDetails(&proto.ErrorDetail{
		Detail: "serve in opposite court",
	})

	if err != nil {
		return nil, status.Error(codes.Internal, "status with details failed")
	}

	return nil, alwaysError(errorStatus)
}

Client Side:

const call = pingPongService.backspin(pingServe, {}, (err: grpcWeb.Error, res: Pong) => {
    console.error(err);
    console.log(res);
});

call.on('status', (status: grpcWeb.Status) => {
    console.log(status);
});

[caseylucas]

Does the typescript definition for Error also need to be changed here (https://github.com/grpc/grpc-web/blob/master/packages/grpc-web/index.d.ts#L51) to include the optional metadata?

[Green7]

I have the same problem. Does anyone know how to decode grpc-status-details-bin header sent by Golang server ? Regarding #667 I have grpcWeb.Error with metadata filed. But unfortunately this field has only "grpc-status" and "grpc-message" headers so I have no way to get content of grpc-status-details-bin. Any hints?

[twixthehero]

I have the same problem. Does anyone know how to decode grpc-status-details-bin header sent by Golang server ? Regarding #667 I have grpcWeb.Error with metadata filed. But unfortunately this field has only "grpc-status" and "grpc-message" headers so I have no way to get content of grpc-status-details-bin. Any hints?

For me, the problem came down to the Envoy proxy not passing grpc-status-details-bin in the response header access-control-expose-headers. I had to expose the grpc-status-details-bin header to allow XHRHttpRequest to read the value (which is used under the hood by XhrIo--see xhrio.js#L1273 and xhrio.js#L96). If you're following the example, on this line https://github.com/grpc/grpc-web/blob/master/net/grpc/gateway/examples/echo/envoy.yaml#L34, add grpc-status-details-bin:

expose_headers: custom-header-1,grpc-status,grpc-message,grpc-status-details-bin

[Green7]

@twixthehero I have grpc-status-details-bin passed by Envoy. The problem is that I don't have idea how to get it from grpc client. Can you send example ?

[twixthehero]

@twixthehero I have grpc-status-details-bin passed by Envoy. The problem is that I don't have idea how to get it from grpc client. Can you send example ?

I used @jscti 's example above like so:

First, I included google's status.proto and any.proto in my protoc compilation. You can download the latest version of status.proto from the googleapis repository. any.proto is included in the protoc downloads here (look in the include/ folder).

Once these are compiled, they are used in my Typescript code like so:

import { DeleteCharacterResponse } from './character_pb.d';
import * as grpcWeb from 'grpc-web';
import { ErrorDetails } from 'src/proto/error/error_details_pb';
import { Status } from 'src/proto/google/rpc/status_pb';

...

const promise = new Promise<DeleteCharacterResponse>((resolve, reject) => {
    const metadata = { Authorization: `Bearer token` };
    // this.service is my grpc client
    const stream = this.service.deleteCharacter(
        request,
        metadata,
        (err: Error | Status, response: DeleteCharacterResponse): void => {
            if (err) {
                reject(err);
            } else {
                resolve(response);
            }
        }
    );

    stream.on('data', (response: DeleteCharacterResponse) => {
        console.log('deleteCharacter data: ' + JSON.stringify(response));
    });

    stream.on('status', (status: grpcWeb.Status) => {
        console.log('deleteCharacter status: ' + JSON.stringify(status));

        const metadata = status.metadata;
        if (metadata !== undefined) {
            const statusEncoded = metadata['grpc-status-details-bin'];
            const statusDecoded = atob(statusEncoded);
            const status = Status.deserializeBinary(
                this.stringToUint8Array(statusDecoded)
            );

            const details = status
              .getDetailsList()
              .map((detail: any) =>
                  ErrorDetails.deserializeBinary(detail.getValue_asU8())
              );

            console.log(`details: ${JSON.stringify(details)}`);
        }
    });

    stream.on('end', () => {
        console.log('deleteCharacter end');
    });

    stream.on('error', (err: grpcWeb.Error) => {
        console.log('deleteCharacter error: ' + JSON.stringify(err));
    });
});

// convert promise to observable and return
const observable = from<Promise<DeleteCharacterResponse>>(promise);
return observable.pipe(...);

...

private stringToUint8Array(str: string): Uint8Array {
  const buf = new ArrayBuffer(str.length);
  const bufView = new Uint8Array(buf);
  for (let i = 0; i < str.length; i++) {
    bufView[i] = str.charCodeAt(i);
  }
  return bufView;
}

ErrorDetails is the proto message that you are adding to the details field. In my case:

syntax = "proto3";

import "google/rpc/error_details.proto";

package error;

message ErrorDetails {
    google.rpc.RetryInfo retryInfo = 1;
    google.rpc.DebugInfo debugInfo = 2;
    google.rpc.QuotaFailure quotaFailure = 3;
    google.rpc.PreconditionFailure preconditionFailure = 4;
    google.rpc.BadRequest badRequest = 5;
    google.rpc.RequestInfo requestInfo = 6;
    google.rpc.Help help = 7;
    google.rpc.LocalizedMessage localizedMessage = 8;
}

[Green7]

@twixthehero Thank you very much. It works ! :thumbsup: