[Enhancement] Optimize creating table using a shared initial tablet metadata between tablets in shared-data mode
Why I'm doing:
Creating table is very slow when there is many tablets in shared-data mode
What I'm doing:
Since the content stored in the tablet metadata of version 1 is the same for all tablets in the same partition, we can let all tablets share the same tablet metadata of version 1. In this way, when creating a table, no matter how many tablets there are in a partition, only one tablet metadata needs to be written, making the time to create a table independent of the number of tablets, speeding up the creation of tables.
Add fe config lake_enable_tablet_creation_optimization to control this optimization.
Fixes https://github.com/StarRocks/StarRocksTest/issues/7246
What type of PR is this:
- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Does this PR entail a change in behavior?
- [ ] Yes, this PR will result in a change in behavior.
- [x] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
- [x] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
- [ ] I have added documentation for my new feature or new function
- [ ] This is a backport pr
Bugfix cherry-pick branch check:
- [x] I have checked the version labels which the pr will be auto-backported to the target branch
- [x] 3.3
- [ ] 3.2
- [ ] 3.1
- [ ] 3.0
- [ ] 2.5
Quality Gate passed
Issues
9 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
Quality Gate passed
Issues
9 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
[FE Incremental Coverage Report]
:white_check_mark: pass : 25 / 29 (86.21%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | com/starrocks/alter/LakeTableSchemaChangeJob.java | 6 | 8 | 75.00% | [334, 406] |
| :large_blue_circle: | com/starrocks/server/LocalMetastore.java | 11 | 13 | 84.62% | [1858, 2005] |
| :large_blue_circle: | com/starrocks/common/Config.java | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | com/starrocks/task/CreateReplicaTask.java | 7 | 7 | 100.00% | [] |
[BE Incremental Coverage Report]
:white_check_mark: pass : 42 / 48 (87.50%)
file detail
| path | covered_line | new_line | coverage | not_covered_line_detail | |
|---|---|---|---|---|---|
| :large_blue_circle: | be/src/storage/task/engine_batch_load_task.cpp | 0 | 1 | 00.00% | [66] |
| :large_blue_circle: | be/src/storage/lake/vacuum.cpp | 2 | 4 | 50.00% | [596, 597] |
| :large_blue_circle: | be/src/storage/lake/metadata_iterator.cpp | 4 | 6 | 66.67% | [33, 34] |
| :large_blue_circle: | be/src/storage/lake/tablet_manager.cpp | 26 | 27 | 96.30% | [445] |
| :large_blue_circle: | be/src/service/service_be/lake_service.cpp | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/storage/lake/filenames.h | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/storage/lake/update_manager.cpp | 1 | 1 | 100.00% | [] |
| :large_blue_circle: | be/src/storage/lake/metadata_iterator.h | 2 | 2 | 100.00% | [] |
| :large_blue_circle: | be/src/storage/lake/transactions.cpp | 3 | 3 | 100.00% | [] |
| :large_blue_circle: | be/src/storage/lake/location_provider.h | 2 | 2 | 100.00% | [] |
@Mergifyio backport branch-3.3
backport branch-3.3
✅ Backports have been created
- #48103 [Enhancement] Optimize creating table using a shared initial tablet metadata between tablets in shared-data mode (backport #45666) has been created for branch
branch-3.3but encountered conflicts