bonsaidb
bonsaidb copied to clipboard
Refactor Views to utilize Nebari's B-Tree
The view-histogram example is a neat example, but the current implementation of views means that there's still a large amount of concatenation of histograms. With our approach to storage in Sled, there was no actual tree to latch onto for the reduce/re-reduce functionality. With Nebari, there will be after:
- khonsulabs/nebari#19
- khonsulabs/nebari#21
- khonsulabs/nebari#23
- khonsulabs/nebari#44
The net effect of this change for the histogram example is that the internal tree would be responsible for reducing the samples, allowing for fewer calculations for any given query.
This task is to refactor the view mapping algorithm to store Vec<ViewEntry> in the tree, and utilize the new functionality from the linked issue to call the view's map and reduce code from within the B-Tree indexing callbacks.
The last detail will be to hook into the node scanning callback in khonsulabs/nebari#21 to gather values without reading data and using the stored information on the index at the highest level possible.
@freopen and I had some productive discussions on Discord regarding the functionality/function signature of reduce. Through it, I realized that we're going to need to want to support Nebari's concept of being able to transition datatypes from the "leaf nodes" to the "interior nodes".
While it's powerful to have the ability to have two associated types, we should ensure it's easy to implement a simplified reduce function when the Value and ReducedValue types are the same, such that the single function is called for both reduce and rereduce operations.