How to dynamically choose output topics

[Tobo7]

Is it possible to dynamically set output topic of StreamingDataFrame pipeline? I need the output topic name to be derived from message content.

Something like this: sdf.to_topic(lambda value: str(value["target_topic"]))

Native Kafka streams API supports it and its the last thing that keeps me away from using this library. https://developer.confluent.io/confluent-tutorials/dynamic-output-topic/kstreams/

Thank you!

[ghost]

Hi @Tobo7 , thanks for creating the issue!

What you're asking about makes sense. I'll see what it would take to implement it.

In your case, do you know the expected number of topics upfront? Or is it completely dynamic?

[Tobo7]

In our case, approximately 50-100 topics, but the number will increase in the future.

We collect all DML changes from all database schemas and their tables into one topic. We need to split the messages into multiple topics per schema... so it's highly dynamic.

Thank you for your interest!

[ghost]

Hi there, we released the support for dynamic topic routing in v3.21.0. Posting it here in case someone is looking for this feature.