flow icon indicating copy to clipboard operation
flow copied to clipboard

Storage for Aggregating Functions

Open norberttech opened this issue 11 months ago • 0 comments

Currently, all Aggregating Functions are storing aggregated results in memory. This is not fully aligned with Flow core philosophy which says that Flow is supposed to be memory efficient in the first place.

To solve that problem we should introduce a dedicated storage for mixed results of aggregating functions into which they can easily read/write to.

It should be configurable, meaning that end user should be able to chose if he wants to aggregate values in memory (super fast but not scalable) or to use one of the available storage strategies.

We can approach it in the similar way to ~GroupBy~ Sort:

  • when memory consumption < X - use memory
  • when memory consumption > X - use cache

We might be better using dedicated storage rather than Cache which is optimized for Rows. In this case, Redis-based storage might be a very good solution.

norberttech avatar Jan 18 '25 03:01 norberttech