starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] support compact spilled mem-table

Open stdpain opened this issue 1 year ago • 3 comments

Why I'm doing:

What I'm doing:

lineorder is from ssb100G

set pipeline_dop=4;
set spill_mode="force";
select max(sm) from (select sum(lo_linenumber) sm from lineorder group by lo_orderkey)x;
sv bytes spilled cost (s) mem peak
disable spill 0 6.54 9.268 GB
mem-table-size=104857600 1.764 GB 16.81 2.098 GB
mem-table-size=10485760 1.408 GB 15.79 178.480 MB
mem-table-size=1048576 2.313 GB 33.39 994.145 MB
mem-table-size=1048576 compact=true 9.062 GB 33.94 60.166 MB

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

stdpain avatar May 07 '24 09:05 stdpain

[FE Incremental Coverage Report]

:white_check_mark: pass : 18 / 18 (100.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/qe/SessionVariable.java 18 18 100.00% []

github-actions[bot] avatar May 28 '24 13:05 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 252 / 254 (99.21%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/exec/pipeline/pipeline_driver_executor.cpp 13 14 92.86% [349]
:large_blue_circle: be/src/exec/spill/input_stream.cpp 63 64 98.44% [257]
:large_blue_circle: be/src/exec/spill/spill_components.h 1 1 100.00% []
:large_blue_circle: be/src/exec/spill/file_block_manager.h 1 1 100.00% []
:large_blue_circle: be/src/exec/spill/spill_components.cpp 73 73 100.00% []
:large_blue_circle: be/src/exec/pipeline/exec_state_reporter.cpp 4 4 100.00% []
:large_blue_circle: be/src/exec/spill/spiller.hpp 2 2 100.00% []
:large_blue_circle: be/src/exec/spill/query_spill_manager.cpp 4 4 100.00% []
:large_blue_circle: be/src/exec/pipeline/sort/spillable_partition_sort_sink_operator.cpp 1 1 100.00% []
:large_blue_circle: be/src/exec/pipeline/aggregate/spillable_aggregate_blocking_sink_operator.cpp 1 1 100.00% []
:large_blue_circle: be/src/exec/spill/spiller.cpp 9 9 100.00% []
:large_blue_circle: be/src/exec/spill/block_manager.h 4 4 100.00% []
:large_blue_circle: be/src/runtime/runtime_state.h 18 18 100.00% []
:large_blue_circle: be/src/exec/spill/log_block_manager.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/spill/spiller.h 1 1 100.00% []
:large_blue_circle: be/src/runtime/runtime_state.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/spill/data_stream.cpp 31 31 100.00% []
:large_blue_circle: be/src/exec/spill/serde.cpp 1 1 100.00% []
:large_blue_circle: be/src/exec/spill/input_stream.h 21 21 100.00% []

github-actions[bot] avatar May 28 '24 14:05 github-actions[bot]