starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Optimize creating table using a shared initial tablet metadata between tablets in shared-data mode

Open xiangguangyxg opened this issue 1 year ago • 4 comments

Why I'm doing:

Creating table is very slow when there is many tablets in shared-data mode

What I'm doing:

Since the content stored in the tablet metadata of version 1 is the same for all tablets in the same partition, we can let all tablets share the same tablet metadata of version 1. In this way, when creating a table, no matter how many tablets there are in a partition, only one tablet metadata needs to be written, making the time to create a table independent of the number of tablets, speeding up the creation of tables.

Add fe config lake_enable_tablet_creation_optimization to control this optimization.

Fixes https://github.com/StarRocks/StarRocksTest/issues/7246

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

xiangguangyxg avatar May 15 '24 08:05 xiangguangyxg

[FE Incremental Coverage Report]

:white_check_mark: pass : 25 / 29 (86.21%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/alter/LakeTableSchemaChangeJob.java 6 8 75.00% [334, 406]
:large_blue_circle: com/starrocks/server/LocalMetastore.java 11 13 84.62% [1858, 2005]
:large_blue_circle: com/starrocks/common/Config.java 1 1 100.00% []
:large_blue_circle: com/starrocks/task/CreateReplicaTask.java 7 7 100.00% []

github-actions[bot] avatar Jul 09 '24 13:07 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 42 / 48 (87.50%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/storage/task/engine_batch_load_task.cpp 0 1 00.00% [66]
:large_blue_circle: be/src/storage/lake/vacuum.cpp 2 4 50.00% [596, 597]
:large_blue_circle: be/src/storage/lake/metadata_iterator.cpp 4 6 66.67% [33, 34]
:large_blue_circle: be/src/storage/lake/tablet_manager.cpp 26 27 96.30% [445]
:large_blue_circle: be/src/service/service_be/lake_service.cpp 1 1 100.00% []
:large_blue_circle: be/src/storage/lake/filenames.h 1 1 100.00% []
:large_blue_circle: be/src/storage/lake/update_manager.cpp 1 1 100.00% []
:large_blue_circle: be/src/storage/lake/metadata_iterator.h 2 2 100.00% []
:large_blue_circle: be/src/storage/lake/transactions.cpp 3 3 100.00% []
:large_blue_circle: be/src/storage/lake/location_provider.h 2 2 100.00% []

github-actions[bot] avatar Jul 09 '24 13:07 github-actions[bot]

@Mergifyio backport branch-3.3

github-actions[bot] avatar Jul 10 '24 06:07 github-actions[bot]

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar Jul 10 '24 06:07 mergify[bot]