starrocks
starrocks copied to clipboard
[Enhancement] [Refactor] Refactor schema scanner & support push predicates into fe for materialized views/task run status
Why I'm doing:
- SchemaTable(information_schema.xxxx) will redirect rpc to FE.
- It's expensive to call rpc(eg: information_schema.task_runs/information_schema.materialized_views) if there are too many meta infos which may cause timeout in rpc.
What I'm doing:
- Introduce
SchemaScannerState
to pass the params used in rpc methods for better extends. - Refactor rpc methods in
SchemaHelper
by usingSchemaScannerState
; - Push down
table_name
forinformation_schema.materialized_views
into FE's rpc to reduce rpc size if query containsTABLE_NAME
equal predicate, eg:TABLE_NAME
= 'mv-1' - Push down
task_name
/query_id
/state
forinformation_schema.task_runs
into FE's rpc to reduce rpc size if query containstask_name
/query_id
/state
equal predicate, eg:task_name
= 'mv-11290' - Fix
test_mv_swap
test.
Fixes #issue
What type of PR is this:
- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [ ] Yes, this PR will result in a change in behavior.
- [x] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [x] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [x] 3.3
- [x] 3.2
- [x] 3.1
- [ ] 3.0
- [ ] 2.5
Quality Gate passed
Issues
10 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
[FE Incremental Coverage Report]
:white_check_mark: pass : 57 / 69 (82.61%)
file detail
path | covered_line | new_line | coverage | not_covered_line_detail | |
---|---|---|---|---|---|
:large_blue_circle: | com/starrocks/scheduler/TaskManager.java | 27 | 34 | 79.41% | [402, 618, 622, 657, 661, 665, 669] |
:large_blue_circle: | com/starrocks/service/FrontendServiceImpl.java | 27 | 32 | 84.38% | [718, 719, 724, 740, 753] |
:large_blue_circle: | com/starrocks/server/LocalMetastore.java | 3 | 3 | 100.00% | [] |
[BE Incremental Coverage Report]
:x: fail : 155 / 220 (70.45%)
file detail
path | covered_line | new_line | coverage | not_covered_line_detail | |
---|---|---|---|---|---|
:large_blue_circle: | be/src/exec/schema_scanner/schema_table_privileges_scanner.cpp | 0 | 2 | 00.00% | [43, 48] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_stream_loads_scanner.cpp | 0 | 2 | 00.00% | [71, 72] |
:large_blue_circle: | be/src/exec/schema_scanner/sys_fe_memory_usage.cpp | 0 | 2 | 00.00% | [41, 47] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_schema_privileges_scanner.cpp | 0 | 2 | 00.00% | [46, 47] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_fe_tablet_schedules_scanner.cpp | 0 | 3 | 00.00% | [74, 75, 76] |
:large_blue_circle: | be/src/exec/schema_scanner/starrocks_role_edges_scanner.cpp | 0 | 2 | 00.00% | [40, 42] |
:large_blue_circle: | be/src/exec/schema_scanner/sys_fe_locks.cpp | 0 | 1 | 00.00% | [51] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_variables_scanner.cpp | 0 | 2 | 00.00% | [60, 61] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_user_privileges_scanner.cpp | 0 | 2 | 00.00% | [45, 46] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_views_scanner.cpp | 0 | 3 | 00.00% | [63, 64, 201] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_routine_load_jobs_scanner.cpp | 0 | 2 | 00.00% | [66, 67] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_helper.cpp | 59 | 95 | 62.11% | [46, 48, 49, 50, 75, 76, 79, 80, 104, 106, 107, 119, 121, 122, 125, 127, 128, 131, 133, 134, 172, 174, 175, 176, 179, 181, 182, 183, 186, 188, 189, 190, 193, 195, 196, 197] |
:large_blue_circle: | be/src/exec/schema_scanner.cpp | 38 | 44 | 86.36% | [102, 324, 325, 326, 327, 352] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_tables_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_partitions_meta_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_pipe_files.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/sys_object_dependencies.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_temp_tables_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_columns_scanner.cpp | 5 | 5 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_task_runs_scanner.cpp | 11 | 11 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_schemata_scanner.cpp | 3 | 3 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_loads_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_tables_config_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_tasks_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/starrocks_grants_to_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner.h | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_materialized_views_scanner.cpp | 6 | 6 | 100.00% | [] |
:large_blue_circle: | be/src/exec/pipeline/scan/schema_scan_context.cpp | 7 | 7 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_pipes.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp | 2 | 2 | 100.00% | [] |
:large_blue_circle: | be/src/exec/schema_scanner/schema_be_tablets_scanner.cpp | 2 | 2 | 100.00% | [] |
@Mergifyio backport branch-3.3
@Mergifyio backport branch-3.2
@Mergifyio backport branch-3.1
backport branch-3.3
✅ Backports have been created
-
#45327 [Enhancement] [Refactor] Refactor schema scanner & support push predicates into fe for materialized views/task run status (backport #44981) has been created for branch
branch-3.3
but encountered conflicts
backport branch-3.2
✅ Backports have been created
-
#45328 [Enhancement] [Refactor] Refactor schema scanner & support push predicates into fe for materialized views/task run status (backport #44981) has been created for branch
branch-3.2
but encountered conflicts
backport branch-3.1
✅ Backports have been created
-
#45329 [Enhancement] [Refactor] Refactor schema scanner & support push predicates into fe for materialized views/task run status (backport #44981) has been created for branch
branch-3.1
but encountered conflicts