confluent-kafka-go
confluent-kafka-go copied to clipboard
confluentinc
Schema Registry implementation for Go
Hi there, I am working for a company that is making extensive use of Golang and Confluent Cloud. I have been working on my own implementation of Schema Registry support for the Go client as this was essential for us. I am wondering if you would appreciate me creating a PR with my Schema Registry support and if so what the best way to submit this would be? For example in a single PR or broken up into chunks? Also so far I have made use of https://github.com/riferrei/srclient for the actual Schema Registry client (which I am also a Contributor on but not the author of). My code is handling the serialisation, wireformat and configuration options. If I submitted this would you want the schema registry client part to be written from scratch in the PR itself or would it be preferable to use this 3rd party lib for this? I am not sure if Confluent wants to use 3rd party libs. Cheers
I also forgot to mention that my implementation targets Protobuf, as that is what most Go programmers use as the lib support for Protobuf is a lot better in Go than Avro support is. This would not prevent Avro serialisers being added later though. The code I have written is closely based on the Python client protobuf serialisation code.
Hi @finncolman , thanks for asking, we are discussing about the schema registry support on go client, will update once we have the decision.
@edenhill , can you please help to answer the question If I submitted this would you want the schema registry client part to be written from scratch in the PR itself or would it be preferable to use this 3rd party lib for this? I am not sure if Confluent wants to use 3rd party libs. ? Thanks.
Hi @jliunyu
Any desicion on adding schema registry support ?
For me that functionality is fundamental, since I am producing messages using this library and consuming the messages using the confluent library for Python.
I am getting this error
"Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer"
Hi @jliunyu Any desicion on adding schema registry support ? For me that functionality is fundamental, since I am producing messages using this library and consuming the messages using the confluent library for Python. I am getting this error
"Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer"
At this moment we don't have a plan to add schema registry support for go client, you can write an SR client or use Ricardo’s and write a wrapper around it and librdkafka.
Hi @jliunyu, the work you described to use Ricardo's client and then put a wrapper around the Go client that uses it and Ricardo's client is exactly what I did in my own implementation. This works fine but it is actually quite a lot of work. In particular the SR client doesn't help do things like get the wire protocol or SERDEs right, which I found required quite a lot of research to get correct. I think the Go client would be a lot more useful with this in place. My implementation closely follows the way it was done for the Python client. I have not modified the existing code for the Go client just added in more code to help with the SERDEs and wire format. I am still wondering if you would prefer a from scratch SR client or Ricardo's though.
Hi @finncolman, sorry for my late reply. I discussed with our team, we recommend to implement this as a separate Go package/module that either wraps our client or is used in combination with it. If things prove useful we can always move them into the client later.
Let me forward the summary related to implementation of an SR client from our team incase some one else has the same question:
1. Implement an SR client (could be based on riferrei's if we think its good, otherwise write one from scratch - we've done a few so far). Then add some kind of serdes API to combine the SR client with Protobuf and our consumers/producers.
2. Use the available community alternatives, namely riferrei's.
LOE:
1. 1+2mo (1 for SR client, 2 for serdes)
2. zero
I created a serdes lib that wraps around the riferrei srclient. It works. I use it at my company. It currently only supports protobuf: GitHub - finncolman/kafka-go-serdes: Serdes for confluent-kafka-go
| | | | | |
|
| | | | GitHub - finncolman/kafka-go-serdes: Serdes for confluent-kafka-go
Serdes for confluent-kafka-go. Contribute to finncolman/kafka-go-serdes development by creating an account on Gi... |
|
|
On Thursday, 3 March 2022, 12:09:34 pm NZDT, Jing Liu ***@***.***> wrote:
Let me forward the summary related to implementation of an SR client from our team incase some one else has the same question:
- implement an SR client (could be based on riferrei's if we think its good, otherwise write one from scratch - we've done a few so far). Then add some kind of serdes API to combine the SR client with Protobuf and our consumers/producers.
- Point people to the available community alternatives, namely riferrei's. LOE:
- 1+2mo (1 for SR client, 2 for serdes)
- zero
— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>
I created a serdes lib that wraps around the riferrei srclient. It works. I use it at my company. It currently only supports protobuf: GitHub - finncolman/kafka-go-serdes: Serdes for confluent-kafka-go
On Tuesday, 12 July 2022, 01:11:21 pm NZST, jflambert ***@***.***> wrote:
Hi, was this issue possibly fixed by @rayokota and #776 ?
I just started using 1.9.1 with schema registry support (JSON Schema) and I'm immediately hit with a "unknown magic byte" error when consuming from a topic.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
