quickwit
quickwit copied to clipboard
Kafka Source: Add support for wildcard or regex-based topic subscription
Is your feature request related to a problem? Please describe. The use-case stems from having a topic per tenant. Currently, I'd have to manually create a source configuration per tenant manually.
Describe the solution you'd like Add support for subscribing to topics via. a regex or wildcard. librdkafka has support for this.
Describe alternatives you've considered Manually adding sources.
Additional context
- https://docs.confluent.io/platform/current/clients/librdkafka/html/rdkafka_8h.html#a0ebe15e9d0f39ccc84e9686f0fcf46f1
- https://discord.com/channels/908281611840282624/915644818988863558/983707895747207211
Although -as discussed on discord- you would like to have 1 index per tenant.