aiokafka icon indicating copy to clipboard operation
aiokafka copied to clipboard

Pass more information to the serializers/deserializers

Open vmaurin opened this issue 1 year ago • 0 comments

Describe the solution you'd like

Serializers and deserializers should receive more context about the record they are working on :

  • the headers
  • the topic

It is especially useful when consuming from multiple topics at the same time, or topic with polymorphic messages advertised in the headers.

All information should be available (example here https://github.com/aio-libs/aiokafka/blob/master/aiokafka/consumer/fetcher.py#L286 and here https://github.com/aio-libs/aiokafka/blob/master/aiokafka/producer/producer.py#L352 ) but achieving backward compatibility could be tricky ? When they added headers to the java interface, they used a "default" method to ensure old implementations will still continue to work

Additional context

Java clients specs : https://kafka.apache.org/28/javadoc/org/apache/kafka/common/serialization/Serializer.html https://kafka.apache.org/28/javadoc/org/apache/kafka/common/serialization/Deserializer.html

vmaurin avatar May 26 '23 08:05 vmaurin