incubator-wayang
incubator-wayang copied to clipboard
Restructure the Kafka related operators
Proposal:
We create a wayang-kafka module inside of wayang-platforms. This leads to a cleaner approach to add these operators.
Right now we edit both wayang-java and wayang-spark, where both of them contain versions of operators related to reading and writing to Kafka.
Conceptually, having all implementations for the specific APIs in wayang-kafka would be neat.
Then you could have wayang-kafka/src/main/java/JavaKafkaOperator wayang-kafka/src/main/java/SparkKafkaOperator etc. and the respective platforms themselves don't have to concern about Kafka.
It is planned to implement this as soon as the initial demos for Kafka integration are ready.
Target: Release 1.1 - Q1-2025.