Dynamic messages using Any

[Shukla-Ankur]

I am building a schema registry and exposing it as a set of APIs. The write API should allow clients to send any message and compare the schema with the stored schema. I tried to use []byte as input type but obviously I lost type information in that case. So I am using Any type now. Following is the strategy I am using:

1. Expose a single API with Any field so that cleints can submit any message
2. For any new message type, clients created .proto and .pb files (for Go), and copy the p.pb file to server also.
3. Server unmarshals the any into a specific type using the .pb file copied by the client.
4. Since all of this needs to be dynamic i.e. new messages can be created at runtime, server does not know the concrete types before-hand.

Server API

message Document {
    DocumentMeta meta = 1;
    bytes data = 2;
    google.protobuf.Any details = 3;
}

On client, I created a new message named Dog and I create the details field as follows:

dog1 := events2.Dog{
    Name: "Leo",
    Id:   1,
}
var dst anypb.Any
err = anypb.MarshalFrom(&dst, &dog1, proto.MarshalOptions{})

The .pb file containing Dog proto is copied in the server also.

If I print the Document in Server, it gives correct information

Any:type_url:"type.googleapis.com/com.test.eventbus.pb.events.Dog" value:"\n\x03Leo\x10\x01"

On Server-side, I am unmarshalling without providing a specific proto.Message type(Dog)

rawData, err := anypb.UnmarshalNew(doc.GetDetails(), proto.UnmarshalOptions{})

The UnmarshalNew fails with proto: not found

I am not providing a specific type during unmarshalling since clients are allowed to send any type of message which may not be defined during application startup time.

How can I use .pb file on server-side while using unmarshalAny() method? Do I need to add the .pb file under a specific path? I could not find a good documentation of its usage

[puellanivis]

The any protobuf has to be able to locate the protobuf definition, not just having the .pb file server-side. How does it know to look for and where to look for it on the server side?

You will need to setup some sort of protoregistry and pass this in as a proto.UnmrashalOptions.Resolver

P.S. as a quick aside, whether you declare a struct with &Struct{} or Struct{} has no bearing on if it’s allocated in heap or in stack, both can be both. So, if you’re only ever using a struct by taking its reference, then it’s a lot simpler to just use the pointer-to-struct literal instead. Especially so in this case, since all generated proto.Messages have pragma.DoNotCopy, so keeping everything in pointers avoids any potential mistaken copying.

[aktau]

At first sight I thought of https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb, but that's more of a solution for unmarshaling and inspecting a proto of which you don't know the type.

If you do have the type information somewhere, you could take the approach proposed by @puellanivis. One of the outputs of the proto compiler is the FileDesciptorProto. The generated .pb.go file should contain a marshaled version of this (the variable name ends in _rawDesc). If your client program was compiled with this .pb.go file, you can use the easier to use protodesc.ToFileDescriptorProto function.

You could transfer this file descriptor proto to the server, and use protodesc.NewFile() on the server to get a protoreflect.FileDescriptor (but make sure the server knows all of the dependencies of said proto file, you must send them in dependency order if the server doesn't have them). This can then be passed to protoregistry.Files.RegisterFile.

[Shukla-Ankur]

I was able to do this by registering MessageTypes. I was not able to register Files since opts.Resolver in anypb.UnmarshalNew is not implemented by Files. So I was not able to give protoregistry.GlobalFiles . However, the new challenge is that the resolver resolves the any field into https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb which I can not use. I was expecting that go would convert the field to concrete type since it has type in Registry now.

[aktau]

I was not able to register Files since opts.Resolver in anypb.UnmarshalNew is not implemented by Files

anypb.UnmarshalNew takes a second opts argument, on which you can set any Resolver: https://github.com/protocolbuffers/protobuf-go/blob/692f4a24f8dc0d375508fc41e657920d411b5b68/types/known/anypb/any.pb.go#L308. The default is protoregistry.GlobalTypes, but you can pass a custom one.

[aktau]

But, looking back at your initial post, it occurs to me that even if you get this to work, it may not be what you want. The reason is that proto unmarshaling is very liberal (especially because it is a best practice to mark fields as optional instead of required). Try it out yourself by marshaling some types and unmarshaling as another type. Quite often there will be no error.

[puellanivis]

However, the new challenge is that the resolver resolves the any field into https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb which I can not use. I was expecting that go would convert the field to concrete type since it has type in Registry now.

Hm… I think in some ways you’re going to be stuck here. There is no way for protobuf to use a concrete type for a protobuf that it only knows the definitions of at runtime. 🤔 In Go, types have to be full defined at compile time, so I’m not sure I could think of any way to work the any protobuf without having to resort to a dynamicpb.

[stapelberg]

In Go, types have to be full defined at compile time, so I’m not sure I could think of any way to work the any protobuf without having to resort to a dynamicpb.

It’s not clear to me from reading this issue whether the .proto files are available at compile-time to link into the server program, or if the server only has access to .proto files at runtime.

@Shukla-Ankur Can you share your code please? (Ideally a minimal standalone example.)