Multi-Stream Consumption

[sajjad-moradi]

Currently one realtime Pinot table only consumes from one stream. For use cases in which multiple existing streams are the input source for one Pinot table, the workaround is to use stream processors to read from different streams, perform necessary transformations, and write the output of all transformed events into another stream. Then one realtime pinot table can be set up to consume from that output stream. This requires maintaining two systems: the stream processor jobs and also pinot realtime tables. It would be nice to add capabilities to pinot realtime tables to consume from multiple topics.

[sajjad-moradi]

I'm working on a design doc for this and will share it here soon.

[icefury71]

@sajjad-moradi this is a great idea. Lemme know if you had any update on this issue.