console
console copied to clipboard
proto: support protobuf any
Better Protobuf Any support
This commit adds better support for protobuf deserialization by allowing the console to deserialize custom proto messages that are supplied to fields with the google.protobuf.Any
type.
Before this change, if your protobuf message contained a field of type google.protobuf.Any
and the value supplied at runtime was not a standard proto message (e.g. google.protobuf.Timestamp
), then msg.MarshalJSON
(called from deserializeProtobufMessageToJSON
) would yield the following error:
failed to marshal protobuf message to JSON:
unknown message type \"<type-url>\"
This error was emitted by the github.com/jhump/protoreflect
because it would fail to resolve the 'type url' of the value supplied to the google.protobuf.Any
field. This error would occur even if the type was available in the proto registry.
The error occurred, because the internal AnyResolver
created by github.com/jhump/protoreflect
only contained the file descriptor relating to the overall protobuf message being processed. Therefore if the type referred to by the 'google.protobuf.Any' field existed in some other file, the type could not be resolved.
This commit amends this behaviour by calling msg.MarshalJSONPB
and passing a custom AnyResolver
through, defined in console/backend/pkg/proto/service.go
. The custom AnyResolver
takes precedence over the internal AnyResolver
, but the internal AnyResolver
is still used by github.com/jhump/protoreflect
if the custom AnyResolver
cannot resolve a type. The custom AnyResolver
provided, strips the "type url" of anything before the last '/' and then calls the proto registry to resolve the type. Removing the contents of the string before the slash is necessary because types begin with "type.googleapis.com/", but the proto registry does not contain this prefix.
The result of the change is that registry types can now be considered, which allows the console to support deserialization of custom proto message types that are supplied to a google.protobuf.Any
field.
WORKING EXAMPLE config:
logger:
level: debug
serveFrontend: false
server:
listenPort: 9090
kafka:
brokers: ["localhost:9092"]
protobuf:
enabled: true
mappings:
- topicName: topic4
valueProtoType: cloud.CloudEvent
git:
enabled: true
refreshInterval: 5m
repository:
url: <path to proto repo>
baseDirectory: proto
basicAuth:
enabled: true
username: <>
password: <>
proto repo:
// cloud/cloud.proto
syntax = "proto3";
package cloud;
option go_package = "./cloud";
import "google/protobuf/timestamp.proto";
import "google/protobuf/any.proto";
message CloudEvent {
string spec_version = 1;
string type = 2;
string source = 3;
string id = 4;
google.protobuf.Timestamp time = 5;
string data_content_type = 6;
string event_version = 7;
google.protobuf.Any data = 8;
}
// user/upload.proto
syntax = "proto3";
package user;
option go_package = "./user";
message UploadEvent {
Properties properties = 1;
string user_id = 2;
repeated string tags = 3;
}
message Properties {
string name = 1;
string email = 2;
string age = 3;
}
result:
🚀 Amazing PR, thank you!!
Pleasure !
what if I have multiple repositories with proto files which Any can reference?