How to send and receive "google.protobuf.Any"?

[maxliu007]

Here is the protobuf message: message MergeResourcesRequest { ResourceType type = 1; repeated google.protobuf.Any resources = 2; }

How can we fire this message to ther server with grpcurl? As it reports follow error: Error invoking method "utsv.event.api.v1.EventService/MergeResources": error getting request data: unknown message type "utsv.event.api.v1.resources.Merchant"

Here ist eh json: { "type" : 15, "resources" : [ { "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant", "id" : 123, "code" : "Merchant", "name": { "en": "desc en", "zh": "desc tw" } } ] }

[jhump]

@maxliu007, are you using server reflection? If so, then if the server knows about that message type, it should work.

If you are not using server reflection, then you must provide -proto or -protoset arguments that indicate not only the source for the service you are using but also the sources for all message types.

If you are using server reflection but it's still not working, then there may be an issue with the server reflection implementation in your server. But a work-around could still be to point grpcurl at local proto source files for the messages. You can do this, while still using server reflection, but providing both -proto arguments (to point to proto sources for messages) and the -use-reflection argument. With this, grpcurl will try server reflection first, to resolve any messages or extensions, and then, failing that, try the proto sources.

[maxliu007]

We are using Java 13 and with server reflection enabled. Tested with following grpc version: <grpc.version>1.33.0</grpc.version> <protoc.version>3.12.0</protoc.version>

When we are testing the non-Any message, grpculr can reply positive result. However it fails whenever we are firing the Any message to the server. From what we observe so far, it has been stucked on the grpcurl. So I wonder if grpcurl can support Any message or not.

Here is the one tested with verbose output:

---

c:\temp\sample\merge-resources-merchant.json"

Resolved method descriptor: rpc MergeResources ( .utsv.event.api.v1.MergeResourcesRequest ) returns ( .utsv.event.api.v1.MergeResourcesResponse );

Request metadata to send: (empty) Error invoking method "utsv.event.api.v1.EventService/MergeResources": error getting request data: unknown message type "utsv.event.api.v1.resources.Merchant"

---

[jhump]

@maxliu007, yes, grpcurl can handle Any messages.

What do you get if you try using grpcurl to invoke the following RPC method, with the following payload?

# add any other necessary parameters to communicate
# with your server, including changing localhost:1234 to
# valid host and port
grpcurl -plaintext \
    -d '{"file_containing_symbol":"utsv.event.api.v1.resources.Merchant"}' \
    localhost:1234 \
    grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo

If that returns an error, such as "not found", that indicates the server is for some reason unable to resolve the message name and provide a descriptor to grpcurl.

Also, as I mentioned in my first reply, you can also try using -use-reflection and -proto arguments, where the -proto arg would point to the proto source file in which utsv.event.api.v1.resources.Merchant is defined.

[maxliu007]

@jhump Here is the result:

---

grpcurl -plaintext -d "{"file_containing_symbol":"utsv.event.api.v1.resources.Merchant"}" 128.128.1.37:9090 grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo { "original_request": { "file_containing_symbol": "utsv.event.api.v1.resources.Merchant" }, "error_response": { "error_code": 5, "error_message": "Symbol not found." } }

---

I found that the reflection cannot resolve any messages that reside in the sub-package under "utsv.event.api.v1", e.g. "utsv.event.api.v1.resources.*". I don't know what is the root clause for such issue.

However, when I tried someting reside inside the package "utsv.event.api.v1", it couldn't decode the response message.

---

grpcurl -plaintext -d "{"file_containing_symbol":"utsv.event.api.v1.TicketType"}" 128.128.1.37:9090 grpc.reflection.v1alpha.ServerReflection.ServerReflectionInfo { "original_request": { "file_containing_symbol": "utsv.event.api.v1.TicketType" }, "file_descriptor_response": { "file_descriptor_proto": [ "Ch9yZXNvdXJjZXMvdGlja2V0X3Jlc29 .....

---

grpcurl -plaintext -d @ 128.128.1.37:9090 utsv.event.api.v1.EventService/MergeResources < "c:\temp\utsv-grpc-sample\merge-resources-ticket-type.json" { "type": "TICKET_TYPE", "resources": [ { "@error": "utsv.event.api.v1.TicketType is not recognized; see @value for raw binary message data", "@type": "type.googleapis.com/utsv.event.api.v1.TicketType", "@value": "CHkSClRpY2tldFR5cGUaDQoCZW4SB2Rlc2MgZW4aDQoCemgSB2Rlc2MgdHciAzAwMQ==" } ] }

[maxliu007]

@jhump After some further tests, I realized that all messages binding to the Any type should be included in the service protobuf file explicitly no matter they are used or not (i.e. no explicit reference in the service protobuf file due to Any which is similar to Object.class in Java). Then we can use server reflection to retrieve the actual type of the Any type, in our case the "utsv.event.api.v1.resources.Merchant". However, we are still facing the decoding problem: { "@error": "utsv.event.api.v1.TicketType is not recognized; see @value for raw binary message data", "@type": "type.googleapis.com/utsv.event.api.v1.TicketType", "@value": "CHkSClRpY2tldFR5cGUaDQoCZW4SB2Rlc2MgZW4aDQoCemgSB2Rlc2MgdHciAzAwMQ==" }

[jhump]

@maxliu007, I would try the same grpcurl invocation to hit the reflection endpoint. If the server is not able to serve the descriptor for utsv.event.api.v1.TicketType, then I'm afraid this error is expected. As I've stated in the past, you can still provide the proto sources to grpcurl directly, so it can resolve these types from source.

After some further tests, I realized that all messages binding to the Any type should be included in the service protobuf file explicitly no matter they are used or not

All language runtimes are a little different, and thus implementations of the reflection service are a little different. I haven't worked with the Java runtime for protobufs in several years, but when I did, I do recall that there was no single registry of all compiled message types. So that might explain why the server cannot "discover" all of these message types unless they are in the transitive closure of the service itself.

[maxliu007]

@jhump I know why the server reflection is not working on utsv.event.api.v1.resources.Merchant now as mentioned above.

For utsv.event.api.v1.TicketType , the server can serve the corresponding descriptor correctly with reflection in fact. However, the grpcurl can't reslove TicketType from the response and reports "@error": "utsv.event.api.v1.TicketType is not recognized; see @value for raw binary message data" instead.

This time I import all Any related to message types to service protobuf explicitly. And I retry the MergeResources test on Merchant, I encounter the same issue.

Using server reflection (import all Any related message types explicitly in service protobuf) - failed to resolve the Any to Merchant

grpcurl -plaintext -d @ 128.128.1.37:9090 utsv.event.api.v1.EventService/MergeResources <  "c:\temp\utsv-grpc-sample\merge-resources-merchant.json"
{
  "type": "MERCHANT",
  "resources": [
    {
      "@error": "utsv.event.api.v1.resources.Merchant is not recognized; see @value for raw binary message data",
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "@value": "CHkSCE1lcmNoYW50Gg0KAmVuEgdkZXNjIGVuGg0KAnpoEgdkZXNjIHR3"
    },
    {
      "@error": "utsv.event.api.v1.resources.Merchant is not recognized; see @value for raw binary message data",
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "@value": "CHoSCE1lcmNoYW50Gg0KAmVuEgdkZXNjIGVuGg0KAnpoEgdkZXNjIHR3"
    },
    {
      "@error": "utsv.event.api.v1.resources.Merchant is not recognized; see @value for raw binary message data",
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "@value": "CHsSCE1lcmNoYW50Gg0KAmVuEgdkZXNjIGVuGg0KAnpoEgdkZXNjIHR3"
    },
    {
      "@error": "utsv.event.api.v1.resources.Merchant is not recognized; see @value for raw binary message data",
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "@value": "CHwSCE1lcmNoYW50Gg0KAmVuEgdkZXNjIGVuGg0KAnpoEgdkZXNjIHR3"
    }
  ]
}

Using protoset - can resolve the Any to Merchant

grpcurl -plaintext -protoset utsv_event.protoset -d @ 128.128.1.37:9090 utsv.event.api.v1.EventService/MergeResources <  "c:\temp\utsv-grpc-sample\merge-resources-merchant.json"
{
  "type": "MERCHANT",
  "resources": [
    {
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "code": "Merchant",
      "id": "121",
      "name": {
        "en": "desc en",
        "zh": "desc tw"
      }
    },
    {
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "code": "Merchant",
      "id": "122",
      "name": {
        "en": "desc en",
        "zh": "desc tw"
      }
    },
    {
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "code": "Merchant",
      "id": "123",
      "name": {
        "en": "desc en",
        "zh": "desc tw"
      }
    },
    {
      "@type": "type.googleapis.com/utsv.event.api.v1.resources.Merchant",
      "code": "Merchant",
      "id": "124",
      "name": {
        "en": "desc en",
        "zh": "desc tw"
      }
    }
  ]
}