druid icon indicating copy to clipboard operation
druid copied to clipboard

Support topicPattern's in kafka ingestion.

Open cryptoe opened this issue 2 years ago • 0 comments

Description

It's about time we support topic patterns in Kafka ingestion. This will enable us to ingest data from multiple topics into a single data source.

Motivation

A lot of companies have the same schema data, from different cluster regions, flowing into different topics with a pattern like topic_cluster_region. Currently, users do a union query over multiple data sources .

Ideally, druid should be able to ingest data from multiple topics into a single data source so that users need not do union on the query side.

Also, I see a lot of people on the druid slack channel requesting this issue.

https://github.com/apache/druid/pull/2656#issuecomment-215328237 This work was also planned as part of the initial Kafka ingestion PR.

cryptoe avatar Oct 13 '22 12:10 cryptoe