cube icon indicating copy to clipboard operation
cube copied to clipboard

Epic: Streaming pre-aggregations

Open rpaik opened this issue 1 year ago • 0 comments

Currently, pre-aggregations are limited to batching data, and it is almost impossible to design efficient pre-aggregations for real-time data.

In the first phase of streaming pre-aggregations we will support kSQL as a data source, and pre-aggregation will subscribe to changes from kSQL. The major difference from traditional pre-aggregations is that streaming pre-aggregations are push-based vs. pull-based. Unlike traditional pre-aggregations, streaming ones do not periodically pull the source database for updates but instead subscribe to the data source to receive updates.

rpaik avatar Aug 01 '22 15:08 rpaik