starrocks
starrocks copied to clipboard
[Enhancement] support memory estimation of array_agg and group_concat state
Why I'm doing:
What I'm doing:
This is a part of #48195 follow-up work, In this PR, I support memory estimation of array_agg and group_concat. These two states are somewhat similar, as they maintain their state internally through Columns.
Currently, the cost of obtaining the memory usage of a certain Column is very high, and we need to balance performance and memory statistics accuracy.
considering that these two functions just aggregates data into Column, does not perform deduplication operations, I directly use the size of the input column to estimate memory changes.
In the future, if our Column becomes an allocator aware structure, we can obtain memory usage at a very low cost.
What type of PR is this:
- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [ ] Yes, this PR will result in a change in behavior.
- [x] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [ ] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [ ] 3.3
- [ ] 3.2
- [ ] 3.1
- [ ] 3.0
- [ ] 2.5
[Java-Extensions Incremental Coverage Report]
:white_check_mark: pass : 0 / 0 (0%)
[FE Incremental Coverage Report]
:white_check_mark: pass : 0 / 0 (0%)
[BE Incremental Coverage Report]
:x: fail : 39 / 56 (69.64%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | be/src/exprs/agg/array_agg.h | 19 | 28 | 67.86% | [192, 227, 228, 229, 232, 233, 234, 236, 265] |
| :large_blue_circle: | be/src/exprs/agg/group_concat.h | 17 | 25 | 68.00% | [315, 329, 330, 331, 332, 333, 336, 394] |
| :large_blue_circle: | be/src/exprs/function_context.h | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | be/src/exec/aggregator.cpp | 1 | 1 | 100.00% | [] |