starrocks
starrocks copied to clipboard
[Enhancement] optimize scheduler one task cost
Why I'm doing:
In our cluster with more than 200 routine load jobs, when I trace com.starrocks.load.routineload.RoutineLoadTaskScheduler:scheduleOneTask, some times get very slow in checkTaskInJob
`---ts=2024-07-30 22:59:08;thread_name=pool-17-thread-47;id=b0a;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@63947c6b
`---[28802.292221ms] com.starrocks.load.routineload.RoutineLoadManager:checkTaskInJob()
+---[0.00% 0.001172ms ] com.starrocks.load.routineload.RoutineLoadManager:readLock() #622
+---[100.00% min=9.12E-4ms,max=626.112241ms,total=28802.075645ms,count=107] com.starrocks.load.routineload.RoutineLoadJob:containsTask() #625
`---[0.00% 0.003156ms ] com.starrocks.load.routineload.RoutineLoadManager:readUnlock() #631
What I'm doing:
Replace RoutineLoadMgr::checkTaskInJob with only check whether task in current job, than check task in all current jobs.
Fixes #issue
What type of PR is this:
- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [x] Yes, this PR will result in a change in behavior.
- [ ] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [x] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [ ] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [x] 3.3
- [x] 3.2
- [x] 3.1
- [x] 3.0
- [x] 2.5
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
[Java-Extensions Incremental Coverage Report]
:white_check_mark: pass : 0 / 0 (0%)
[FE Incremental Coverage Report]
:white_check_mark: pass : 4 / 4 (100.00%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | com/starrocks/load/routineload/RoutineLoadTaskScheduler.java | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/load/routineload/RoutineLoadMgr.java | 3 | 3 | 100.00% | [] |
[BE Incremental Coverage Report]
:white_check_mark: pass : 0 / 0 (0%)
@Mergifyio backport branch-3.3
@Mergifyio backport branch-3.2
@Mergifyio backport branch-3.1
backport branch-3.3
✅ Backports have been created
- #53179 [Enhancement] optimize scheduler one task cost (backport #50328) has been created for branch
branch-3.3
backport branch-3.2
✅ Backports have been created
- #53180 [Enhancement] optimize scheduler one task cost (backport #50328) has been created for branch
branch-3.2
backport branch-3.1
✅ Backports have been created
- #53181 [Enhancement] optimize scheduler one task cost (backport #50328) has been created for branch
branch-3.1