[Enhancement] adjust the BE and CN schedule policy
Why I'm doing: In shared nothing mode,when deploy both BE and CN,and set prefer_compute_node=true,current BE and CN schedule policy is not very feel perfect: 1)when query internal table ,except scan fragment allocate to BE,other calculate fragment would allocate to CN 2)when query external table ,all fragment would allocate to CN
In some scenarios, like when sudden business peak, need temporarily add CN to increase computational resources, current BE and CN schedule policy would cause BE nodes have low resource usage ratio,CN nodes have high resource usage ratio,especially when business type belongs to computational tasks.
What I'm doing: Add a new session variable computation_fragment_scheduling_policy, the value as follows: 1)compute_nodes_only:the previous policy, as the default value 2)all_nodes: a.when query internal table ,scan fragment allocate to BE,other calculate fragment would allocate to both BE and CN b.when query external table ,all fragment would allocate to both BE and CN
Fixes #issue
What type of PR is this:
- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [x] Yes, this PR will result in a change in behavior.
- [ ] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [x] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [x] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [ ] 3.3
- [x] 3.2
- [ ] 3.1
- [ ] 3.0
- [ ] 2.5
@kevincai Hi could you please approval the workflow? the pr is from the previous(38498), i update and reopen it.
@kevincai it looks like fail at SQL-Tester 's test_object_dependencies, Could you please check the Checker? i think the pr have little relation with object_dependencies.
@ZiheLiu Hi could you please review the pr? the pr is from the previous(#38498 ), i update and reopen it.
@kevincai Hi Could you please review the pr when have free time? In our prod env, we use the be/cn schedule policy in the pr,also i want listen to the community's suggestion for feasibility of the schedule policy.
@kevincai Could you please approve the workflow again? i fix a problem(It failed in part of AnalyzeSetVariableTest.java in FE UT. Becase it not contain IllegalArgumentException in the analyzeFail function. i add it) yesterday night.
Quality Gate passed
Issues
4 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
[FE Incremental Coverage Report]
:white_check_mark: pass : 36 / 45 (80.00%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | com/starrocks/qe/SessionVariable.java | 5 | 13 | 38.46% | [2259, 2260, 2261, 2262, 2263, 2264, 2266, 2267] |
| :large_blue_circle: | com/starrocks/qe/scheduler/DefaultWorkerProvider.java | 16 | 17 | 94.12% | [348] |
| :large_blue_circle: | com/starrocks/qe/SessionVariableConstants.java | 3 | 3 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/sql/analyzer/SetStmtAnalyzer.java | 8 | 8 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/qe/CoordinatorPreprocessor.java | 4 | 4 | 100.00% | [] |
[BE Incremental Coverage Report]
:white_check_mark: pass : 0 / 0 (0%)
@kevincai It looks like need [StarRocks/analyzer] to review also, could you please help to look for related classmate to review ?
@Mergifyio backport branch-3.3
@Mergifyio backport branch-3.2
backport branch-3.3
✅ Backports have been created
- #45681 [Enhancement] adjust the BE and CN schedule policy (backport #44499) has been created for branch
branch-3.3
backport branch-3.2
✅ Backports have been created
- #45682 [Enhancement] adjust the BE and CN schedule policy (backport #44499) has been created for branch
branch-3.2but encountered conflicts