starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Hang on show routine load stmt

Open rickif opened this issue 3 years ago • 4 comments
trafficstars

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Which issues of this PR fixes :

Fixes #https://github.com/StarRocks/starrocks/issues/9778

Problem Summary(Required) :

Sometimes, Kafka's response may be slow. When the routine load job is scheduled, the partition info needs to be checked. The previous implementation gets Kafka partition info with the write lock, while the show routine load statement needs read lock, which may lead to the hang of the statement. This PR introduces two modifications:

  1. getting Kafka info with a read lock.
  2. some refactoring to simplify the code.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] I have added user document for my new feature or new function

rickif avatar Sep 16 '22 14:09 rickif

run starrocks_fe_unittest

rickif avatar Sep 17 '22 03:09 rickif

[FE PR Coverage Check]

:disappointed: fail : 24 / 39 (61.54%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/load/routineload/KafkaRoutineLoadJob.java 9 23 39.13% [297, 298, 299, 300, 301, 302, 303, 307, 311, 313, 322, 323, 324, 325]
:large_blue_circle: com/starrocks/load/routineload/RoutineLoadJob.java 15 16 93.75% [1241]

wanpengfei-git avatar Sep 17 '22 04:09 wanpengfei-git

run starrocks_fe_unittest

rickif avatar Sep 20 '22 02:09 rickif

@mergifyio rebase

rickif avatar Sep 29 '22 13:09 rickif

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/3)
error: could not apply 9478272d8... fix: hang on show routine load stmt
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 9478272d8... fix: hang on show routine load stmt
Auto-merging fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java
CONFLICT (content): Merge conflict in fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java

err-code: 25FBF

mergify[bot] avatar Sep 29 '22 13:09 mergify[bot]

run starrocks_fe_unittest

rickif avatar Sep 30 '22 02:09 rickif

@Mergifyio rebase

rickif avatar Nov 11 '22 01:11 rickif

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/3)
error: could not apply 9478272d8... fix: hang on show routine load stmt
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 9478272d8... fix: hang on show routine load stmt
Auto-merging fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java
CONFLICT (content): Merge conflict in fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java
Auto-merging fe/fe-core/src/main/java/com/starrocks/load/routineload/KafkaRoutineLoadJob.java

err-code: E7516

mergify[bot] avatar Nov 11 '22 01:11 mergify[bot]