databend icon indicating copy to clipboard operation
databend copied to clipboard

feat: introduce columnar segment info cache

Open SkyFan2002 opened this issue 5 months ago โ€ข 0 comments

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

The test scenario

  • 3000 segments
  • each segment contains 1000 block-metas
  • the table comprises 25 columns
    • 20 of them are of type String the min/max values of them in block meta, are set to be 16 chars (which is the max number of chars that will be kept)
    • 5 of them are of type Int

Cache SegmentInfo:

scenario 3000 SegmentInfo, 1000 block per seg , pid 2688957, base memory 224849920

            cache item type : SegmentInfoCache,
            number of cached items 3000,
            mem usage(B):+26773348352,
            mem usage(GB)+24.93462371826172

Cache ColumnarSegmentInfo:

scenario 3000 SegmentInfo, 1000 block per seg , pid 2688450, base memory 225886208

            cache item type : ColumnarSegmentInfoCache,
            number of cached items 3000,
            mem usage(B):+8549457920,
            mem usage(GB)+7.962303161621094

Tests

  • [ ] Unit Test
  • [ ] Logic Test
  • [x] Benchmark Test
  • [ ] No Test - Explain why

Type of change

  • [ ] Bug Fix (non-breaking change which fixes an issue)
  • [x] New Feature (non-breaking change which adds functionality)
  • [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • [ ] Documentation Update
  • [ ] Refactoring
  • [ ] Performance Improvement
  • [ ] Other (please describe):

This change isโ€‚Reviewable

SkyFan2002 avatar Sep 24 '24 05:09 SkyFan2002