starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] support memory estimation of array_agg and group_concat state

Open silverbullet233 opened this issue 1 year ago • 3 comments
trafficstars

Why I'm doing:

What I'm doing:

This is a part of #48195 follow-up work, In this PR, I support memory estimation of array_agg and group_concat. These two states are somewhat similar, as they maintain their state internally through Columns.

Currently, the cost of obtaining the memory usage of a certain Column is very high, and we need to balance performance and memory statistics accuracy.

considering that these two functions just aggregates data into Column, does not perform deduplication operations, I directly use the size of the input column to estimate memory changes.

In the future, if our Column becomes an allocator aware structure, we can obtain memory usage at a very low cost.

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

silverbullet233 avatar Jul 17 '24 07:07 silverbullet233

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 06 '24 05:09 github-actions[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 06 '24 05:09 github-actions[bot]

[BE Incremental Coverage Report]

:x: fail : 39 / 56 (69.64%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/exprs/agg/array_agg.h 19 28 67.86% [192, 227, 228, 229, 232, 233, 234, 236, 265]
:large_blue_circle: be/src/exprs/agg/group_concat.h 17 25 68.00% [315, 329, 330, 331, 332, 333, 336, 394]
:large_blue_circle: be/src/exprs/function_context.h 2 2 100.00% []
:large_blue_circle: be/src/exec/aggregator.cpp 1 1 100.00% []

github-actions[bot] avatar Sep 06 '24 05:09 github-actions[bot]