starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] limit tablet write request size

Open silverbullet233 opened this issue 1 year ago • 3 comments

Why I'm doing:

When there are large chunks during the load process, the following error may be returned due to exceeding the size limit of protobuf. img_v3_02e4_eddde0ab-db20-4044-99a3-d6c6a82baf6g

NodeChannel currently only aggregates batches based on the function of the chunk, but does not take into account the size of the chunk. img_v3_02e4_9f4f7683-ef85-49db-b315-5a3647cb863g

What I'm doing:

In this PR, I have added chunk mem usage check to avoid generating large pb requests for load tasks

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

silverbullet233 avatar Aug 27 '24 05:08 silverbullet233

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 19 '24 11:09 github-actions[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 19 '24 11:09 github-actions[bot]

[BE Incremental Coverage Report]

:x: fail : 0 / 24 (00.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/exec/tablet_sink_index_channel.cpp 0 24 00.00% [418, 419, 420, 421, 422, 423, 425, 426, 427, 428, 429, 430, 440, 453, 462, 470, 471, 481, 502, 514, 523, 524, 534, 589]

github-actions[bot] avatar Sep 19 '24 11:09 github-actions[bot]

@Mergifyio backport branch-3.3

github-actions[bot] avatar Sep 19 '24 11:09 github-actions[bot]

@Mergifyio backport branch-3.2

github-actions[bot] avatar Sep 19 '24 11:09 github-actions[bot]

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar Sep 19 '24 11:09 mergify[bot]

backport branch-3.2

✅ Backports have been created

mergify[bot] avatar Sep 19 '24 11:09 mergify[bot]

Can we backport it to 3.2? Thank you!

dengliu avatar Sep 19 '24 23:09 dengliu

Can we backport it to 3.2? Thank you!

I think so, you can cherry-pick https://github.com/StarRocks/starrocks/pull/51173 @dengliu

silverbullet233 avatar Sep 20 '24 10:09 silverbullet233