numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

Support for mulitiple kafka topics by regexp

Open th0ger opened this issue 8 months ago • 0 comments

Summary

Some kafka clients allows regexp patterns for the topic name to consume from multiple topics.

Is this possible with the sarama client? Or can this be supported?

Here from the Confluent kafka python Consumer:

subscribe(topics[, on_assign=None][, on_revoke=None][, on_lost=None]) Set subscription to supplied list of topics. (...) Regexp pattern subscriptions are supported by prefixing the topic string with "^", e.g.: consumer.subscribe(["^my_topic.*", "^another[0-9]-?[a-z]+$", "not_a_regex"])

Parameters topics (list(str)) – List of topics (strings) to subscribe to. (...)


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

th0ger avatar Jun 13 '24 13:06 th0ger