feat(compaction): limit memory usage for compaction read

[Little-Wallace]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

LRU-Cache can not limit memory strictly, although we can make insert LRU-Cache fail when memory-limit full, but it's difficult to make write request block until memory enough.

So I used an independent sstable-store and memory-limiter to limit memory usage

Checklist

• [ ] I have written necessary rustdoc comments
• [ ] I have added necessary unit tests and integration tests
• [ ] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

[codecov[bot]]

Codecov Report

Merging #4590 (fdb328a) into main (0d920b4) will increase coverage by 0.01%. The diff coverage is 74.09%.

@@            Coverage Diff             @@
##             main    #4590      +/-   ##
==========================================
+ Coverage   74.05%   74.06%   +0.01%     
==========================================
  Files         862      863       +1     
  Lines      130243   130524     +281     
==========================================
+ Hits        96448    96671     +223     
- Misses      33795    33853      +58     

Flag	Coverage Δ
rust	74.06% <74.09%> (+0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/compute/src/server.rs	0.00% <0.00%> (ø)
src/storage/compactor/src/server.rs	0.00% <0.00%> (ø)
src/storage/src/hummock/compactor/context.rs	0.00% <0.00%> (ø)
src/storage/src/hummock/iterator/backward_user.rs	96.78% <ø> (-0.01%)	:arrow_down:
src/storage/src/hummock/iterator/mod.rs	59.78% <ø> (ø)
src/storage/src/hummock/mod.rs	85.58% <ø> (ø)
src/storage/src/hummock/test_utils.rs	97.61% <ø> (ø)
src/storage/src/lib.rs	100.00% <ø> (ø)
src/storage/src/monitor/state_store_metrics.rs	84.14% <0.00%> (+0.42%)	:arrow_up:
src/storage/src/storage_failpoints/test_sstable.rs	96.55% <ø> (ø)
... and 21 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Little-Wallace]

This PR could reduce half of memory but it can not still explain why our system use much more memory than statistic.

main branch:

current branch:

[fuyufjh]

Question: is it possible to eliminate caching on the compactor? I don't think caching is helpful here, because every SST being read will be compacted into another SST and the previous files won't be read (compacted) any more.

[Little-Wallace]

Question: is it possible to eliminate caching on the compactor? I don't think caching is helpful here, because every SST being read will be compacted into another SST and the previous files won't be read (compacted) any more.

Yes. I would not cache any data in compactor any more in this PR. I only cache meta-file during compaction job just for several sub-compaction job share the same meta-file. I do not use LRU-Cache for data file and each sub-compaction just read their own data, which may be some range of a data file.

[Little-Wallace]

@hzxa21 please take a look again