Decoding protobuf response

[essequie1]

I recently switched up from protobuf-js to this library, i really liked the change and it is working fine. The thing is that there was a chrome extension i used to check the requests being made in every client of the application, and it doesn't work with the clients generated by protobuf-ts.

I thought about making an extension myself as i didn't believe it would be that hard, and i actually was able to get the response body from all the requests from the clients i generated in a chrome extension i made, when tried decoding them from base64 to string it gives a string with non ascii characters and numbers in between the real data of the response.

This is the one of the respones i got:

const test = "AAAAAH4KIwgCEgRUZXN0GhdodHRwczovL3d3dy5nb29nbGUuY29tLyACChwIBRIFU3BlY3MaDy9zcGVjaWZpY2F0aW9ucyABCjkIBxIDV0lQGi4vc3BlY2lmaWNhdGlvbnMvc2VhcmNoP3N0YXR1cz1Xb3JrK0luK1Byb2dyZXNzIAE=gAAAABBncnBjLXN0YXR1czogMA0K"

Buffer.from(test, "base64").toString()

When decoded gives me this: '~ #Testhttps://www.google.com/ Specs/specifications 9WIP./specifications/search?status=Work+In+Progress '

This is what the "grcp web dev tools" gives me:

{
  grpc: {
    method: '...url/GetFavorites',
    request: {},
    response: {
      favoritesList: [
        {
          id: 2,
          label: 'Test',
          url: 'https://www.google.com/',
          favoriteType: 2,
        },
        {
          id: 5,
          label: 'Specs',
          url: '/specifications',
          favoriteType: 1,
        },
        {
          id: 7,
          label: 'WIP',
          url: '/specifications/search?status=Work+In+Progress',
          favoriteType: 1,
        },
      ],
    },
  },
};

I tried in many ways but i didn't found that much information about it in the internet, so i came here to ask if anyone knows how can i decode the information in the response to a JSON or JS Object or at least a simple string?

Thanks!

[timostamm]

gRPC-web uses a message framing. It's superficially documented here. It's just length delimited, with space for a couple of flags.

You can take a look at the transport implementation here if you want to roll your own:

https://github.com/timostamm/protobuf-ts/blob/3a7ce47d43113d1a80cacd0ae70630b3727eda3e/packages/grpcweb-transport/src/grpc-web-format.ts#L156

[essequie1]

Is the first part of that function just waiting for a request and then creates an Uint8Array from it, right? I'm guessing i can just convert the base64 string from a response into an Uint8Array and pass it to a function that just does this:

function readGrpcWebResponseBody(uintArr, onFrame) {
  let byteQueue = uintArr;
  while (byteQueue.length >= 5 && byteQueue[0] === GrpcWebFrame.DATA) {
    let msgLen = 0;

    for (let i = 1; i < 5; i++) {
      msgLen = (msgLen << 8) + byteQueue[i];
    }

    if (byteQueue.length - 5 >= msgLen) {
      // we have the entire message
      onFrame(GrpcWebFrame.DATA, byteQueue.subarray(5, 5 + msgLen));
      byteQueue = byteQueue.subarray(5 + msgLen);
    } else {
      break;
    }
  }
}

What i don't understand is how onFrame works, because it takes the "O" from the method of the response and uses .fromBinary() passing the actual response as a parameter?

Sorry for the misunderstanding but i don't really know that much about this and i can't actually wrap my mind around how this works.

PS.: How the "grpc-web-dev-tools" works is that it adds interceptors to all the clients and decodes or captures the responses from them. I don't really want to do that. What i'm doing right now is just capture the request responses when they finish and i don't really know if the information i got from the response is enought to actually decode the messages.

[timostamm]

gRPC-web uses a message framing. The function splits an incoming stream of binary data into individual frames.

i don't really know if the information i got from the response is enought to actually decode the messages.

To split the frames, yes. To parse protobuf messages from binary, no. See https://protobuf.dev/programming-guides/encoding/

[jcready]

You can get the "grpc-web-dev-tools" extension working with protobuf-ts by adding this interceptor to your grpc-web transport:

import type { RpcInterceptor } from '@protobuf-ts/runtime-rpc';
import { RpcError } from '@protobuf-ts/runtime-rpc';

const type = '__GRPCWEB_DEVTOOLS__';

export const grpcWebDevToolsInterceptor: RpcInterceptor = {
    interceptUnary(next, method, input, options) {
        const res = next(method, input, options);
        // @ts-ignore
        if (window.__GRPCWEB_DEVTOOLS__) {
            const methodType = 'unary';
            const m = `${method.service.typeName}/${method.name}`;
            const request = method.I.toJson(res.request);
            void res.then((value) => {
                window.postMessage({
                    type,
                    method: m,
                    methodType,
                    request,
                    response: method.O.toJson(value.response),
                });
            }).catch((e) => {
                if (e instanceof RpcError) {
                    window.postMessage({
                        type,
                        method: m,
                        methodType,
                        request,
                        error: {
                            code: e.code,
                            message: e.message,
                        },
                    });
                }
            });
        }
        return res;
    },
    interceptServerStreaming(next, method, input, options) {
        const res = next(method, input, options);
        // @ts-ignore
        if (window.__GRPCWEB_DEVTOOLS__) {
            const m = `${method.service.typeName}/${method.name}`;
            const methodType = 'server_streaming';
            window.postMessage({
                type,
                method: m,
                methodType,
                request: method.I.toJson(res.request),
            });
            res.responses.onMessage((value) => {
                window.postMessage({
                    type,
                    method: m,
                    methodType,
                    response: method.O.toJson(value),
                });
            });
            res.responses.onError((e) => {
                if (e instanceof RpcError) {
                    window.postMessage({
                        type,
                        method: m,
                        methodType,
                        error: {
                            code: e.code,
                            message: e.message,
                        },
                    });
                }
            });
            res.responses.onComplete(() => {
                window.postMessage({
                    type,
                    method: m,
                    methodType,
                    response: 'EOF',
                });
            });
        }
        return res;
    },
};

[essequie1]

You can get the "grpc-web-dev-tools" extension working with protobuf-ts by adding this interceptor to your grpc-web transport:

Does this have an impact in performance? Thank you anyways!

[jcready]

Does this have an impact in performance?

Only if you have the "grpc-web-dev-tools" extension enabled.

[glmnet]

@jcready I'm using the code you posted and is failing for stream because m is no longer the method name on the onMessage callback

[jcready]

Ah, sorry about that. I believe I've fixed the code above.