[Enhancement] Optimize large tablet lake compaction by support parallel compaction

[meegoo]

Why I'm doing:

What I'm doing:

Fixes #issue

What type of PR is this:

• [ ] BugFix
• [ ] Feature
• [x] Enhancement
• [ ] Refactor
• [ ] UT
• [ ] Doc
• [ ] Tool

Does this PR entail a change in behavior?

• [ ] Yes, this PR will result in a change in behavior.
• [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
• [ ] Parameter changes: default values, similar parameters but with different default values
• [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
• [ ] Feature removed
• [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• [x] I have added test cases for my bug fix or my new feature
• [ ] This pr needs user documentation (for new or modified features or behaviors)
  • [ ] I have added documentation for my new feature or new function
• [ ] This is a backport pr

Bugfix cherry-pick branch check:

• [x] I have checked the version labels which the pr will be auto-backported to the target branch
  • [ ] 4.0
  • [ ] 3.5
  • [ ] 3.4
  • [ ] 3.3

---

[!NOTE] Introduces per-tablet parallel compaction with subtask scheduling/merging, partial compaction handling, new configs, and proto/FE wiring.

• Backend (Lake compaction):
  • Parallel compaction: Add TabletParallelCompactionManager and integrate with CompactionScheduler to split a tablet’s compaction into parallel subtasks, merge txn logs, and fall back to serial when needed.
  • Policy/Execution: CompactionPolicy::pick_rowsets_with_limit(...) for byte-limited picks and exclude sets; TabletManager::compact(...) overload accepts pre-selected rowsets.
  • Partial compaction support: Add PartialCompactionSelector and state helpers to detect partial completion and prepare force-publish.
  • Build: Include new source in be/src/storage/CMakeLists.txt.
• FE (Scheduler/Config):
  • Request wiring: FE sets visibleVersion, table/partition ids, autonomous mode flag, and optional per-tablet parallel config on CompactRequest (both normal and aggregate paths).
  • Configs: Add toggles and limits for autonomous/parallel compaction and partial publish.
• Protobuf/API:
  • Add TabletParallelConfig; extend CompactRequest with table_id, partition_id, autonomous_compaction, visible_version, and parallel_config; extend CompactResponse with subtask_statuses.
  • Add CompactionResultPB to lake_types.proto.
• Configs (BE):
  • Add autonomous compaction, per-tablet parallel limits, and recovery/partial publish settings in config.h.

^{Written by Cursor Bugbot for commit 0b34040bdc2d61d0aa8d2a79f18d3f9faf780ee7. This will update automatically on new commits. Configure here.}

[mergify[bot]]

🧪 CI Insights

Here's what we observed from your CI run for 0b34040b.

🟢 All jobs passed!

But CI Insights is watching 👀

[alvin-celerdata]

@cursor review

[alvin-celerdata]

@cursor review

[alvin-celerdata]

@cursor review

[alvin-celerdata]

@cursor review

[github-actions[bot]]

[BE Incremental Coverage Report]

:x: fail : 119 / 544 (21.88%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	be/src/storage/lake/tablet_parallel_compaction_manager.h	0	5	00.00%	[39, 48, 85, 144, 145]
:large_blue_circle:	be/src/storage/lake/tablet_writer.h	0	1	00.00%	[161]
:large_blue_circle:	be/src/storage/lake/tablet_parallel_compaction_manager.cpp	2	312	00.64%	[43, 49, 50, 51, 52, 54, 57, 58, 62, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 77, 78, 79, 82, 83, 84, 85, 86, 88, 89, 92, 94, 95, 96, 97, 99, 100, 106, 107, 113, 114, 115, 116, 117, 118, 122, 123, 124, 127, 128, 129, 130, 131, 132, 133, 134, 135, 137, 138, 141, 142, 143, 144, 146, 148, 149, 150, 151, 152, 154, 155, 156, 158, 159, 161, 163, 164, 165, 167, 168, 169, 172, 173, 175, 176, 178, 179, 180, 181, 182, 185, 186, 188, 190, 192, 193, 194, 195, 196, 197, 198, 199, 201, 202, 204, 205, 207, 208, 210, 211, 212, 213, 214, 218, 221, 222, 224, 225, 227, 228, 230, 231, 232, 233, 236, 239, 240, 241, 244, 245, 246, 248, 252, 253, 254, 255, 256, 259, 261, 264, 265, 266, 268, 269, 270, 273, 274, 275, 277, 278, 279, 280, 281, 282, 284, 285, 286, 287, 290, 291, 292, 294, 296, 301, 302, 303, 310, 311, 312, 313, 314, 320, 321, 326, 327, 328, 329, 330, 332, 333, 334, 335, 336, 337, 340, 341, 345, 346, 350, 351, 355, 356, 359, 360, 370, 371, 372, 374, 378, 379, 380, 381, 382, 386, 387, 388, 389, 394, 395, 396, 397, 398, 399, 400, 402, 403, 405, 411, 413, 414, 416, 420, 421, 423, 424, 428, 431, 432, 434, 437, 438, 440, 442, 445, 446, 448, 450, 453, 454, 457, 458, 464, 465, 466, 467, 474, 475, 477, 478, 481, 482, 483, 484, 485, 486, 487, 490, 493, 495, 496, 498, 499, 500, 503, 505, 506, 508, 509, 510, 511, 513, 514, 517, 520, 521, 523, 526, 527, 528, 529, 530, 533, 534, 537, 538, 541, 542, 543, 547, 548, 550, 551, 552, 553, 556, 557, 558, 559, 562, 566, 567, 568, 569, 571, 572, 573, 576, 577]
:large_blue_circle:	be/src/storage/lake/compaction_task_context.h	0	5	00.00%	[84, 89, 90, 91, 92]
:large_blue_circle:	be/src/storage/rows_mapper.cpp	11	69	15.94%	[140, 141, 142, 143, 145, 154, 163, 164, 165, 169, 172, 173, 174, 176, 177, 178, 181, 182, 184, 186, 189, 190, 192, 194, 198, 199, 200, 201, 204, 205, 206, 208, 209, 210, 211, 212, 213, 216, 217, 220, 221, 222, 225, 226, 228, 229, 230, 233, 234, 235, 236, 237, 238, 239, 241, 242, 244, 245]
:large_blue_circle:	be/src/storage/lake/compaction_scheduler.cpp	24	57	42.11%	[297, 298, 299, 300, 320, 321, 322, 323, 325, 326, 328, 329, 331, 333, 334, 335, 336, 338, 340, 341, 342, 343, 345, 346, 347, 348, 350, 351, 352, 353, 355, 356, 357]
:large_blue_circle:	be/src/storage/lake/compaction_task.cpp	1	2	50.00%	[47]
:large_blue_circle:	be/src/storage/lake/horizontal_compaction_task.cpp	1	2	50.00%	[71]
:large_blue_circle:	be/src/storage/lake/vertical_compaction_task.cpp	1	2	50.00%	[64]
:large_blue_circle:	be/src/storage/lake/update_manager.cpp	7	14	50.00%	[1225, 1226, 1227, 1230, 1231, 1232, 1233]
:large_blue_circle:	be/src/storage/lake/pk_tablet_writer.cpp	18	20	90.00%	[53, 180]
:large_blue_circle:	be/src/storage/lake/compaction_policy.cpp	9	10	90.00%	[443]
:large_blue_circle:	be/src/storage/lake/primary_key_compaction_policy.h	14	14	100.00%	[]
:large_blue_circle:	be/src/storage/lake/primary_key_compaction_policy.cpp	9	9	100.00%	[]
:large_blue_circle:	be/src/storage/lake/tablet_manager.cpp	22	22	100.00%	[]

[alvin-celerdata]

@cursor review

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
C Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[github-actions[bot]]

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

[github-actions[bot]]

[FE Incremental Coverage Report]

:x: fail : 7 / 21 (33.33%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	com/starrocks/service/InformationSchemaDataSource.java	0	3	00.00%	[480, 481, 484]
:large_blue_circle:	com/starrocks/lake/compaction/CompactionScheduler.java	2	13	15.38%	[411, 412, 413, 414, 415, 416, 457, 458, 459, 460, 461]
:large_blue_circle:	com/starrocks/common/Config.java	5	5	100.00%	[]

[alvin-celerdata]

@cursor review