Deserialize Protobuf payload based on 4-byte Schema ID from Schema Registry

[ericsun2]

Support for Protobuf with Confluent Schema Registry If the Kafka event is serialized in Protobuf and a 4-byte integer schema_id is put in front of the Protobuf binary stream, can kafka-go invoke the REST API of Confluent Schema Registry to fetch the .proto schema text from Schema Registry, and then form a decoder/deserializer on-the-fly to deserialize the Protobuf binary stream?

Supporting documentation

• https://docs.confluent.io/1.0/schema-registry/docs/api.html#schemas
• https://docs.confluent.io/platform/current/schema-registry/serdes-develop/serdes-protobuf.html
• https://www.confluent.io/blog/using-protobuf-in-confluent-cloud/#protobuf-with-cli-tools

[jandyalaphani]

Any update on this

[jordaniversen]

Poke

[achille-roussel]

Hello, thanks for starting this conversation.

kafka-go exposes the content of messages as an opaque byte array, the package currently does not unmarshal nor interpret the data.

From where we stand as maintainers of kafka-go, this functionality appears to deserve a dedicated package since it does not have a tight dependency to kafka-go itself, any byte array could be decoded from schemas fetched from remote locations.

We would happily provide visibility to such package in kafka-go, or potentially host it as a submodule if we have a convincing case for it. At this time such a change will need to come from a community contributor, the core maintainer team will not be able to allocate development time to build this feature.

Let me know what you think!