WIP: Avro/Schema-Registry

[rnpridgeon]

This is a request for a review/comment of a WIP Schema Registry implementation.

I have attempted to stick to the Java API as much as possible with some minor changes but I've run into a bit of a mental block on deserialization. I'm not sure how to best handle deserialization into specific types.

I have toyed with a few implementations but I think they too are less than ideal

1. take a new()interface{} function pointer upon instantiating the deserializer. This is similar to the way sync.Pool works

2. Move Avro serialization/deserialization out of the Producer/Consumer opting instead to have a stateful marshaler/unmarshaler.

3. The current implementation which takes an instance on instantiation and makes a zero copy then places the results in the Message.Object field which was added in the serde_support development branch to facilitate handling objects for serialization/deserialization.

In the end I suspect the best way to move forward is option 2. This puts serialization/deserialization outside of the producer/consumer however which may not be what we want.

All relevant code is located in the encoding/avro folder. Specifically serializer.go

[ghost]

@confluentinc It looks like @rnpridgeon just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

[ccl0326]

@edenhill Sorry, any progress now for this pr?

[ccl0326]

Hi, @rnpridgeon . I don't know if you hive time to finish this pull request, if not, i think i can continue to finish it if you allow.

[kairotavares]

Any updates on the Schema Registry support?

[tven-aspiration]

Hoping this gets merged in soon. All our services are written in Go and we need this to publish topics with registry support.

[udam6688]

any updates on avro and serialization support?

[wshirey]

@edenhill are the issues you raised the only thing stopping this from being merged in? I could try and fix those changes if @rnpridgeon is not willing to see this PR through. We are looking at adopting Confluent Cloud as an enterprise option and I'm sure sales and Go shops would appreciate this being merged in.

[cchexcode]

Is there any progress on this? Right now this really prevents some people on using Go in combination with Confluent Kafka... And in a Go environment, this can be a deal breaker. Nevertheless, there is still some work to do but essentially we are more than 1 year behind the original plan for supporting the schema registry which was Q2/early Q3 2018 (see #69).

[edenhill]

The current plan is to finalize generic serializer and schema registry support in Q1. As the Avro situation in Go is a bit messy we'll not provide builtin Avro serdes, but provide examples how to implement your own using one of the popular Avro alternatives.

[cchexcode]

@edenhill Ok I understand. Sounds like a good plan to me. Q1 is too far away for immediate resolution some problems but I guess it's worth to spend more time on this feature when looking at the importance of it. Let me know if I can support by reviewing or similar, I am happy to help when I can. Does there already exist an issue with this particular proposal? I was not able to find one.

[rnpridgeon]

We will rebase and clean uo this PR by the end of the week (hooefully Wednesday or Thursday possible it wont be until Friday though) now. Any reviews after rebasing on master and clearing out the cob webs from before this was deprioritized will be addressed and we'd love a secondary review.

[nitishpt]

Any updates on this? Somewhere above Q1 was mentioned. Is it still the same or postponed? Thanks!

[tarunrathor-pro]

Any update on when this will be done. Schema registry support with avro

[alokkx]

Hi Matt, We are hitting issue of magic byte not being sent for the messages produced by kafka serializer. I see that you have tried to fix it here. Is it fixed now? Is there any workaround.

org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

[aren89]

The current plan is to finalize generic serializer and schema registry support in Q1. As the Avro situation in Go is a bit messy we'll not provide builtin Avro serdes, but provide examples how to implement your own using one of the popular Avro alternatives.

Any update?

[edenhill]

This will not be addressed in Q2, but hopefully in Q3.

[shivakumarss]

Any updates on this

[edenhill]

With a bit of luck we'll revisit this in Q4.

[maurolscla]

Any updates?

[huoarter]

Any updates?

[marcosdotps]

Hello, Please provide info about this feature, if it has been abandoned I would like to reboot it.

[edenhill]

Hi @mpenate, we unfortunately don't have any immediate plans to finish this PR, but if you're up for rebooting it we would do our best to support you in the process!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}