firebolt icon indicating copy to clipboard operation
firebolt copied to clipboard

Is it possible to have source from more than one topic?

Open marshallharry opened this issue 3 years ago • 2 comments

Its not really an issue but I'm wondering if is it possible to have multiple kafka topic in the source with the same node processing the message? Thank you.

marshallharry avatar Jul 08 '21 04:07 marshallharry

@jnadler @cristovarghese please advice, sir

marshallharry avatar Jul 08 '21 04:07 marshallharry

No, multiple topics are not supported today. However this would not be too hard to add to firebolt, because confluent-kafka-go natively supports subscribing to multiple topics at once. It would involve changes in kafkaconsumer.go and recoveryconsumer.go at a minimum. Perhaps we'd expect a CSV string of topics, and split the string into the array used to configure the consumers.

jnadler avatar Jul 08 '21 16:07 jnadler