ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

EPIC: Column stats

Open alexowens90 opened this issue 8 months ago • 0 comments

Known remaining work. Replace checkbox text with link to individual issues when they are being actively worked on.

  • [ ] Use column stats to eliminate unnecessary IOs with the QueryBuilder. Should include logging how many data segment IOs were bypassed due to the stats.
  • [ ] Add option to bypass column stats on queries, as they are user generated and so they may just add unnecessary overhead. Default should be to read them though, as the overhead should be low.
  • [ ] Maintain existing column stats when calling write/append/update. Append and update can reuse existing stats for unchanged row-slices. Stats should also be maintained through write_metadata calls.
  • [ ] New stats (currently only min/max implemented):
    • [ ] Is sorted
    • [ ] Bloom filters
    • [ ] Explicit set membership
  • [ ] Profiling and optimisation

alexowens90 avatar May 30 '24 10:05 alexowens90