Description

We are trying to implement the scenario described here: https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/

We are using the most current Confluent platform.

We discovered two things that are not currently support by the client library:

1. Schema references when registering a schema with type Avro (CachedSchemaRegistryClient).
2. Looking up schema's using the latest version instead of by id (AvroSerializer)

Will there be any support for this scenario anytime soon? We have created a workaround by making a custom AvroSerializer and RestService implementation.

1. That is not well supported by the .NET Avro library to my knowledge - how are you doing it? Note that I would recommend Protobuf over Avro in .NET if you have flexibility to change (the implementation is far more mature).
2. CachedSchemaRegistryClient has GetLatestSchemaAsync - what are you trying to do exactly?

Let me pull your codebase and make a working example for you. I'll get back to you as soon as possible. Should I do this as a PR, branch, fork or something else?

whatever is quick

Hi Matt

Sorry for the delay - back from vacation now.

I created a fork since I didn't have necessary permissions to create a pull request.

https://github.com/Elpaggio/confluent-kafka-dotnet

The fun part is in the "ProduceMultipleEventTypesToSameTopic.cs" file.

1. That is not well supported by the .NET Avro library to my knowledge - how are you doing it? **Note that I would recommend Protobuf over Avro in .NET if you have flexibility to change (the implementation is far more mature).**

@mhowlett can you elaborate more on your comment above advocating ProtoBuf over Avro for .NET?

Unfortunately Avro is governed company wide and has to be used as the preffered schema type across all microservices.

tir. 20. okt. 2020 10.20 skrev Chris Smith [email protected]:

1. That is not well supported by the .NET Avro library to my knowledge - how are you doing it? Note that I would recommend Protobuf over Avro in .NET if you have flexibility to change (the implementation is far more mature).

@mhowlett https://github.com/mhowlett can you elaborate more on your comment above advocating ProtoBuf over Avro for .NET?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/confluentinc/confluent-kafka-dotnet/issues/1425#issuecomment-712681223, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMWLSFLILQQPWWFHTXLBDTSLVB5ZANCNFSM4SLSV2GQ .

What about JSON? @mhowlett @Elpaggio

I have 3 events called UserCreated, UserUpdated and UserDeleted.

I use the same topic called user_events for all types.

But it looks like .net client doesn't support multiple event types in the same topic for json?

What about JSON? @mhowlett @Elpaggio

I have 3 events called UserCreated, UserUpdated and UserDeleted.

I use the same topic called user_events for all types.

But it looks like .net client doesn't support multiple event types in the same topic for json?

@suadev I haven't tried using multiple event types using json. But I think it should work as far as I remember the code. Have you tried?

@mhowlett

Hi Matt

Sorry for the delay - back from vacation now.

I created a fork since I didn't have necessary permissions to create a pull request.

https://github.com/Elpaggio/confluent-kafka-dotnet

The fun part is in the "ProduceMultipleEventTypesToSameTopic.cs" file.

@mhowlett Have you had the time to explore the code changes in the fork?

What about JSON? @mhowlett @Elpaggio I have 3 events called UserCreated, UserUpdated and UserDeleted. I use the same topic called user_events for all types. But it looks like .net client doesn't support multiple event types in the same topic for json?

@suadev I haven't tried using multiple event types using json. But I think it should work as far as I remember the code. Have you tried?

No, it doesn't work, or I am missing something. Do you have a working sample?

@mhowlett are you still interested in this topic?

Unfortunately we'll have to continue using our own fork.

What about JSON? @mhowlett @Elpaggio I have 3 events called UserCreated, UserUpdated and UserDeleted. I use the same topic called user_events for all types. But it looks like .net client doesn't support multiple event types in the same topic for json?

@suadev I haven't tried using multiple event types using json. But I think it should work as far as I remember the code. Have you tried?

No, it doesn't work, or I am missing something. Do you have a working sample?

@suadev you could take a look at our fork and maybe implement the same behaviour we have implemented for the Avro De/Serializers for the json ones and see if it works.

This is a topic I'm interested in also. I've prototyped some custom serializers / deserializers (Avro and Protobuf) to support multiple value types on a single Kafka topic.

This is available at: https://github.com/danmalcolm/confluent-kafka-dotnet/tree/multiple-message-types-on-topic (see the AvroMultipleTypesPerTopic and ProtobufMultipleTypesPerTopic example projects).

This works as follows:

1. At start up, we configure our multiple type serializers / deserializers with a list of the message value types we want to support
2. The multiple type serializer contains a map of standard schema registry serializers (e.g. Confluent.SchemaRegistry.Serdes.AvroSerializer)
3. During serialization, we simply delegate to the serializer for the value being stored
4. During deserialization, we take advantage of the fact that the schema registry serializers include a schema identifier within the data written for each key and value. We look up and delegate to a deserializer that supports the destination value type.

I've no idea whether an approach like this would ever make it back into Confluent.Kafka, where the API is very much focused on specific key and value types.

This is a POC only - no unit tests etc. Hopefully there's enough here for teams to get started if they do want to do this right now in their applications.

@Elpaggio - your work is really useful as you've looked at saving the union of schemas to the topic in the schema registry. I wanted to extend the work I have done to include this functionality. Did you also look at supporting deserialization?

@mhowlett , @Elpaggio I'm also looking for the same and looks like there is no support of reference schema while Deserializing avro message. could you please confirm ? If there is support then please provide some sample code. Thanks

There is same issue with #2168