druid
druid copied to clipboard
Support topicPattern's in kafka ingestion.
Description
It's about time we support topic patterns in Kafka ingestion. This will enable us to ingest data from multiple topics into a single data source.
Motivation
A lot of companies have the same schema data, from different cluster regions, flowing into different topics with a pattern like topic_cluster_region. Currently, users do a union query over multiple data sources .
Ideally, druid should be able to ingest data from multiple topics into a single data source so that users need not do union on the query side.
Also, I see a lot of people on the druid slack channel requesting this issue.
https://github.com/apache/druid/pull/2656#issuecomment-215328237 This work was also planned as part of the initial Kafka ingestion PR.