console icon indicating copy to clipboard operation
console copied to clipboard

proto: support protobuf any

Open chris-palmer-deltatre opened this issue 2 years ago • 1 comments

Better Protobuf Any support

This commit adds better support for protobuf deserialization by allowing the console to deserialize custom proto messages that are supplied to fields with the google.protobuf.Any type.

Before this change, if your protobuf message contained a field of type google.protobuf.Any and the value supplied at runtime was not a standard proto message (e.g. google.protobuf.Timestamp), then msg.MarshalJSON (called from deserializeProtobufMessageToJSON) would yield the following error:

failed to marshal protobuf message to JSON:
unknown message type \"<type-url>\"

This error was emitted by the github.com/jhump/protoreflect because it would fail to resolve the 'type url' of the value supplied to the google.protobuf.Any field. This error would occur even if the type was available in the proto registry.

The error occurred, because the internal AnyResolver created by github.com/jhump/protoreflect only contained the file descriptor relating to the overall protobuf message being processed. Therefore if the type referred to by the 'google.protobuf.Any' field existed in some other file, the type could not be resolved.

This commit amends this behaviour by calling msg.MarshalJSONPB and passing a custom AnyResolver through, defined in console/backend/pkg/proto/service.go. The custom AnyResolver takes precedence over the internal AnyResolver, but the internal AnyResolver is still used by github.com/jhump/protoreflect if the custom AnyResolver cannot resolve a type. The custom AnyResolver provided, strips the "type url" of anything before the last '/' and then calls the proto registry to resolve the type. Removing the contents of the string before the slash is necessary because types begin with "type.googleapis.com/", but the proto registry does not contain this prefix.

The result of the change is that registry types can now be considered, which allows the console to support deserialization of custom proto message types that are supplied to a google.protobuf.Any field.

WORKING EXAMPLE config:

logger: 
  level: debug
serveFrontend: false
server:
  listenPort: 9090
kafka:
  brokers: ["localhost:9092"]
  protobuf:
    enabled: true
    mappings:
      - topicName: topic4
        valueProtoType: cloud.CloudEvent
    git:
      enabled: true
      refreshInterval: 5m
      repository:
        url: <path to proto repo>
        baseDirectory: proto
      basicAuth:
        enabled: true
        username: <>
        password: <>

proto repo:

// cloud/cloud.proto
syntax = "proto3";

package cloud;
option go_package = "./cloud";

import "google/protobuf/timestamp.proto";
import "google/protobuf/any.proto";

message CloudEvent {
    string spec_version = 1;
    string type = 2;
    string source = 3;
    string id = 4;
    google.protobuf.Timestamp time = 5;
    string data_content_type = 6;
    string event_version = 7;
    google.protobuf.Any data = 8;
}
// user/upload.proto
syntax = "proto3";
package user;
option go_package = "./user";

message UploadEvent {
    Properties properties = 1;
    string user_id                    = 2;
    repeated string tags        = 3;
}

message Properties {
   string name = 1;
   string email = 2;
   string age = 3;
}

result: image

chris-palmer-deltatre avatar Aug 10 '22 06:08 chris-palmer-deltatre

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Aug 10 '22 06:08 CLAassistant

🚀 Amazing PR, thank you!!

weeco avatar Aug 25 '22 14:08 weeco

Pleasure !

chris-palmer-deltatre avatar Aug 28 '22 23:08 chris-palmer-deltatre

what if I have multiple repositories with proto files which Any can reference?

sashayakovtseva avatar Apr 05 '23 10:04 sashayakovtseva