ArcticDB
ArcticDB copied to clipboard
EPIC: Column stats
Known remaining work. Replace checkbox text with link to individual issues when they are being actively worked on.
- [ ] Use column stats to eliminate unnecessary IOs with the
QueryBuilder
. Should include logging how many data segment IOs were bypassed due to the stats. - [ ] Add option to bypass column stats on queries, as they are user generated and so they may just add unnecessary overhead. Default should be to read them though, as the overhead should be low.
- [ ] Maintain existing column stats when calling
write/append/update
. Append and update can reuse existing stats for unchanged row-slices. Stats should also be maintained throughwrite_metadata
calls. - [ ] New stats (currently only min/max implemented):
- [ ] Is sorted
- [ ] Bloom filters
- [ ] Explicit set membership
- [ ] Profiling and optimisation