flink-connector-kafka icon indicating copy to clipboard operation
flink-connector-kafka copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-XXX] Add set_deserializer method to Python KafkaSourceBuilder

Open fqaiser94 opened this issue 10 months ago • 1 comments

Haven't been able to file a JIRA ticket yet as I'm waiting on account creation.

Currently PyFlink users have access to only the set_value_only_deserializer API for configuring how to deserialize Kafka messages. This API is extremely limited as it does not allow PyFlink users to:

  1. Deserialize keys
  2. Access ConsumerRecord metadata such as topic, partition, offset, etc.

We can resolve this by exposing a set_deserializer method which will allow users to pass in a KafkaRecordDeserializationSchema object that describes how to deserialize a ConsumerRecord. This is similar to the pattern followed by the Java implementation of KafkaSourceBuilder.

fqaiser94 avatar Jan 13 '25 19:01 fqaiser94

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

boring-cyborg[bot] avatar Jan 13 '25 19:01 boring-cyborg[bot]