registry
registry copied to clipboard
hortonworks
Schema Registry implementation for Protobuf desired
Due to gRPC's popularity in many embedded systems, Schema Registry for Protobuf is highly desired, which is also already claimed by Hortonworks as an advantage over other choice (http://registry-project.readthedocs.io/en/latest/competition.html)
@xufuture protobuf encoded messages contain information about schema and it may not require for users to pass schema version ids etc with the payload. IMHO, SchemaRegistry is still useful for protobuf in schema discovery and respective serializer/deserializer. (disclaimer: I am not an expert in protobuf and its APIs) What are your usecases for protobuf support in SchemaRegistry?
@satishd protobuf does not contain schema in its encoded message, you require generated class from IDL to decode it. although it requires code generation(compilation of IDL) before its use, If hortonworks schema registry can provide support to store and check backward/forward compatibility it would be very useful as @xufuture mentioned.
@satishd @raju-saravanan @arunmahadevan we are planning to add protobuf support. created #721 can u guys pls provide comments and support or direct to right engineers who can help us review? Thanks
I think using the json representation of the schema e.g. the proto variant would be better. As like done with avro. The protobuf json schema is more workable when doing bits in ui etc.
Need to ensure the confluent sr compatibility api works with protobuf in same was confluents sr does to maintain compatibility.
Regarding JSON, yes, that would an easy way, but it changes the schema language. we can probaly have both. would start with JSON for now. Thanks for the suggestion.
Dont forget about compatibility layer we have with confluents which theire supports proto now. As they provide many kafka clients for other languages such as .net c++ etc and thus makes ability to use those with this registry.
Sure, I may have to take some iterative way of adding all these, as I am only adding the basic protobuf support first. but I will make sure we dont break any exisiting feature like the one u just mentioned - compatibility with confluent api.
The reason i flag that is that its important from the outset is incase the way the confluent api and proto clients handle it differ to the way you initally thought to do then you will have big compatibility issues later. Which unless designed in at first it will be problematic