databend
databend copied to clipboard
feat: introduce columnar segment info cache
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
The test scenario
- 3000 segments
- each segment contains 1000 block-metas
- the table comprises 25 columns
- 20 of them are of type
String
the min/max values of them in block meta, are set to be 16 chars (which is the max number of chars that will be kept) - 5 of them are of type
Int
- 20 of them are of type
Cache SegmentInfo
:
scenario 3000 SegmentInfo, 1000 block per seg , pid 2688957, base memory 224849920
cache item type : SegmentInfoCache,
number of cached items 3000,
mem usage(B):+26773348352,
mem usage(GB)+24.93462371826172
Cache ColumnarSegmentInfo
:
scenario 3000 SegmentInfo, 1000 block per seg , pid 2688450, base memory 225886208
cache item type : ColumnarSegmentInfoCache,
number of cached items 3000,
mem usage(B):+8549457920,
mem usage(GB)+7.962303161621094
Tests
- [ ] Unit Test
- [ ] Logic Test
- [x] Benchmark Test
- [ ] No Test - Explain why
Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):