ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Make cross-symbol merge work properly

Open willdealtry opened this issue 1 year ago • 0 comments
trafficstars

Merging by time (time-union) of symbols is implemented and has some tests, however it's not currently in a state where we can recommend its use as a first-class feature. In particular it probably needs testing with dynamic schema and sparse data.

It should be quite an easy win in that there's no particular reason to believe that it's fundamentally broken - most of the code involved is used and tested elsewhere, since it uses the processing pipeline to perform the different steps, and the aggregation is the same as streaming data collection.

For the time being the split stage can simply be removed as it's only useful with spill-to-storage, otherwise all the intermediate results need to fit in memory anyway.

willdealtry avatar Apr 18 '24 10:04 willdealtry