kafka-ui
kafka-ui copied to clipboard
provectus
Add support for deserializer plugins
I'm curious if you are open to supporting deserializer plugins (perhaps using the Java Service Provider Interface) to allow third parties to supply their own deserializers.
Use case
We're currently using Kafka UI and our topics contain protobuf binary serialized messages. Our protobuf are stored in the Buf Schema Registry and we've forked Kafka UI to add support for deserialization of protobuf messages using schema information from the Buf Schema Registry.
We would like to continue to benefit from updates to Kafka UI, and also contribute our support for the Buf Schema Registry to the open-source community. However, we're not sure if adding support for the Buf Schema Registry to Kafka UI directly makes sense. In addition, there may be additional cases where others have their own deserializer that they want to plug into Kafka UI, and adding each case to Kafka UI would likely bloat the core application.
This led us to think it may be nice for Kafka UI to support deserialization plugins that anyone can easily make.
Implementation ideas
In terms of implementation, I believe the RecordSerDe interface would need to be stabilized, and plugins would essentially write deserialization implementations that conform to the interface.
In addition, I think it might be useful if RecordSerDe contained a canDeserialize(String topic, ConsumerRecord<Bytes, Bytes> msg) function to check if a deserializer can deserialize a given message. In such a setup, DeserializationService (or some other component) would fall back to other deserializes if needed (ultimately falling back to a simple string deserializer).
Happy to help add plugin support if you think it's a good idea.
Hey, thanks for the interest in contributing!
Let me give a thought to it, I'll get back to you soon.
UPD: We're gonna discuss the subject on Monday. UPD: We're working on describing requirements for the task.
@jdechicchis Hey, feel free to join us in discussing this thing in #1816. We'll finalize some stuff this week.
https://github.com/provectus/kafka-ui/pull/2125/files#diff-b4ae01ceb9a305876d57249a1cd7d5bf7567fd37c5a96cb4e8a4327c8d25b500 kafka-ui-contract/src/main/resources/swagger/kafka-ui-api.yaml To do: Reflect API changes on frontend