starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] [Refactor] Refactor schema scanner & support push predicates into fe for materialized views/task run status

Open LiShuMing opened this issue 9 months ago • 3 comments

Why I'm doing:

  • SchemaTable(information_schema.xxxx) will redirect rpc to FE.
  • It's expensive to call rpc(eg: information_schema.task_runs/information_schema.materialized_views) if there are too many meta infos which may cause timeout in rpc.

What I'm doing:

  • Introduce SchemaScannerState to pass the params used in rpc methods for better extends.
  • Refactor rpc methods in SchemaHelper by using SchemaScannerState;
  • Push down table_name for information_schema.materialized_views into FE's rpc to reduce rpc size if query contains TABLE_NAME equal predicate, eg: TABLE_NAME = 'mv-1'
  • Push down task_name/query_id/state for information_schema.task_runs into FE's rpc to reduce rpc size if query contains task_name/query_id/state equal predicate, eg: task_name = 'mv-11290'
  • Fix test_mv_swap test.

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [x] 3.2
    • [x] 3.1
    • [ ] 3.0
    • [ ] 2.5

LiShuMing avatar Apr 29 '24 11:04 LiShuMing

[FE Incremental Coverage Report]

:white_check_mark: pass : 57 / 69 (82.61%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/scheduler/TaskManager.java 27 34 79.41% [402, 618, 622, 657, 661, 665, 669]
:large_blue_circle: com/starrocks/service/FrontendServiceImpl.java 27 32 84.38% [718, 719, 724, 740, 753]
:large_blue_circle: com/starrocks/server/LocalMetastore.java 3 3 100.00% []

github-actions[bot] avatar May 08 '24 09:05 github-actions[bot]

[BE Incremental Coverage Report]

:x: fail : 155 / 220 (70.45%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/exec/schema_scanner/schema_table_privileges_scanner.cpp 0 2 00.00% [43, 48]
:large_blue_circle: be/src/exec/schema_scanner/schema_stream_loads_scanner.cpp 0 2 00.00% [71, 72]
:large_blue_circle: be/src/exec/schema_scanner/sys_fe_memory_usage.cpp 0 2 00.00% [41, 47]
:large_blue_circle: be/src/exec/schema_scanner/schema_schema_privileges_scanner.cpp 0 2 00.00% [46, 47]
:large_blue_circle: be/src/exec/schema_scanner/schema_fe_tablet_schedules_scanner.cpp 0 3 00.00% [74, 75, 76]
:large_blue_circle: be/src/exec/schema_scanner/starrocks_role_edges_scanner.cpp 0 2 00.00% [40, 42]
:large_blue_circle: be/src/exec/schema_scanner/sys_fe_locks.cpp 0 1 00.00% [51]
:large_blue_circle: be/src/exec/schema_scanner/schema_variables_scanner.cpp 0 2 00.00% [60, 61]
:large_blue_circle: be/src/exec/schema_scanner/schema_user_privileges_scanner.cpp 0 2 00.00% [45, 46]
:large_blue_circle: be/src/exec/schema_scanner/schema_views_scanner.cpp 0 3 00.00% [63, 64, 201]
:large_blue_circle: be/src/exec/schema_scanner/schema_routine_load_jobs_scanner.cpp 0 2 00.00% [66, 67]
:large_blue_circle: be/src/exec/schema_scanner/schema_helper.cpp 59 95 62.11% [46, 48, 49, 50, 75, 76, 79, 80, 104, 106, 107, 119, 121, 122, 125, 127, 128, 131, 133, 134, 172, 174, 175, 176, 179, 181, 182, 183, 186, 188, 189, 190, 193, 195, 196, 197]
:large_blue_circle: be/src/exec/schema_scanner.cpp 38 44 86.36% [102, 324, 325, 326, 327, 352]
:large_blue_circle: be/src/exec/schema_scanner/schema_tables_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_partitions_meta_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_pipe_files.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/sys_object_dependencies.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_temp_tables_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_columns_scanner.cpp 5 5 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_task_runs_scanner.cpp 11 11 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_schemata_scanner.cpp 3 3 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_loads_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_tables_config_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_tasks_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/starrocks_grants_to_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner.h 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_materialized_views_scanner.cpp 6 6 100.00% []
:large_blue_circle: be/src/exec/pipeline/scan/schema_scan_context.cpp 7 7 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_pipes.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp 2 2 100.00% []
:large_blue_circle: be/src/exec/schema_scanner/schema_be_tablets_scanner.cpp 2 2 100.00% []

github-actions[bot] avatar May 08 '24 09:05 github-actions[bot]

@Mergifyio backport branch-3.3

github-actions[bot] avatar May 09 '24 03:05 github-actions[bot]

@Mergifyio backport branch-3.2

github-actions[bot] avatar May 09 '24 03:05 github-actions[bot]

@Mergifyio backport branch-3.1

github-actions[bot] avatar May 09 '24 03:05 github-actions[bot]

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar May 09 '24 03:05 mergify[bot]

backport branch-3.2

✅ Backports have been created

mergify[bot] avatar May 09 '24 03:05 mergify[bot]

backport branch-3.1

✅ Backports have been created

mergify[bot] avatar May 09 '24 03:05 mergify[bot]