quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Kafka Source: Add support for wildcard or regex-based topic subscription

Open godwhoa opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe. The use-case stems from having a topic per tenant. Currently, I'd have to manually create a source configuration per tenant manually.

Describe the solution you'd like Add support for subscribing to topics via. a regex or wildcard. librdkafka has support for this.

Describe alternatives you've considered Manually adding sources.

Additional context

  • https://docs.confluent.io/platform/current/clients/librdkafka/html/rdkafka_8h.html#a0ebe15e9d0f39ccc84e9686f0fcf46f1
  • https://discord.com/channels/908281611840282624/915644818988863558/983707895747207211

godwhoa avatar Jun 07 '22 14:06 godwhoa

Although -as discussed on discord- you would like to have 1 index per tenant.

fulmicoton avatar Jun 09 '22 02:06 fulmicoton