starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] adjust the BE and CN schedule policy

Open MatthewH00 opened this issue 2 months ago • 7 comments

Why I'm doing: In shared nothing mode,when deploy both BE and CN,and set prefer_compute_node=true,current BE and CN schedule policy is not very feel perfect: 1)when query internal table ,except scan fragment allocate to BE,other calculate fragment would allocate to CN 2)when query external table ,all fragment would allocate to CN

In some scenarios, like when sudden business peak, need temporarily add CN to increase computational resources, current BE and CN schedule policy would cause BE nodes have low resource usage ratio,CN nodes have high resource usage ratio,especially when business type belongs to computational tasks.

What I'm doing: Add a new session variable computation_fragment_scheduling_policy, the value as follows: 1)compute_nodes_only:the previous policy, as the default value 2)all_nodes: a.when query internal table ,scan fragment allocate to BE,other calculate fragment would allocate to both BE and CN b.when query external table ,all fragment would allocate to both BE and CN

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [x] Yes, this PR will result in a change in behavior.
  • [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [x] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.3
    • [x] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

MatthewH00 avatar Apr 22 '24 01:04 MatthewH00