starrocks
starrocks copied to clipboard
[Enhancement] support compact spilled mem-table
Why I'm doing:
What I'm doing:
lineorder is from ssb100G
set pipeline_dop=4;
set spill_mode="force";
select max(sm) from (select sum(lo_linenumber) sm from lineorder group by lo_orderkey)x;
| sv | bytes spilled | cost (s) | mem peak |
|---|---|---|---|
| disable spill | 0 | 6.54 | 9.268 GB |
| mem-table-size=104857600 | 1.764 GB | 16.81 | 2.098 GB |
| mem-table-size=10485760 | 1.408 GB | 15.79 | 178.480 MB |
| mem-table-size=1048576 | 2.313 GB | 33.39 | 994.145 MB |
| mem-table-size=1048576 compact=true | 9.062 GB | 33.94 | 60.166 MB |
Fixes #issue
What type of PR is this:
- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [ ] Yes, this PR will result in a change in behavior.
- [x] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [ ] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [ ] 3.3
- [ ] 3.2
- [ ] 3.1
- [ ] 3.0
- [ ] 2.5
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
[FE Incremental Coverage Report]
:white_check_mark: pass : 18 / 18 (100.00%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | com/starrocks/qe/SessionVariable.java | 18 | 18 | 100.00% | [] |
[BE Incremental Coverage Report]
:white_check_mark: pass : 252 / 254 (99.21%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | be/src/exec/pipeline/pipeline_driver_executor.cpp | 13 | 14 | 92.86% | [349] |
| :large_blue_circle: | be/src/exec/spill/input_stream.cpp | 63 | 64 | 98.44% | [257] |
| :large_blue_circle: | be/src/exec/spill/spill_components.h | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/file_block_manager.h | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/spill_components.cpp | 73 | 73 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/pipeline/exec_state_reporter.cpp | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/spiller.hpp | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/query_spill_manager.cpp | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/pipeline/sort/spillable_partition_sort_sink_operator.cpp | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/pipeline/aggregate/spillable_aggregate_blocking_sink_operator.cpp | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/spiller.cpp | 9 | 9 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/block_manager.h | 4 | 4 | 100.00% | [] |
| :large_blue_circle: | be/src/runtime/runtime_state.h | 18 | 18 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/log_block_manager.cpp | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/spiller.h | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/runtime/runtime_state.cpp | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/data_stream.cpp | 31 | 31 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/serde.cpp | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/spill/input_stream.h | 21 | 21 | 100.00% | [] |