incubator-wayang icon indicating copy to clipboard operation
incubator-wayang copied to clipboard

Restructure the Kafka related operators

Open kamir opened this issue 5 months ago • 2 comments

Proposal:

We create a wayang-kafka module inside of wayang-platforms. This leads to a cleaner approach to add these operators.

Right now we edit both wayang-java and wayang-spark, where both of them contain versions of operators related to reading and writing to Kafka.

Conceptually, having all implementations for the specific APIs in wayang-kafka would be neat.

Then you could have wayang-kafka/src/main/java/JavaKafkaOperator wayang-kafka/src/main/java/SparkKafkaOperator etc. and the respective platforms themselves don't have to concern about Kafka.

It is planned to implement this as soon as the initial demos for Kafka integration are ready.

Target: Release 1.1 - Q1-2025.

kamir avatar Sep 05 '24 09:09 kamir