[Enhancement] Support online optimize table

[meegoo]

Why I'm doing:

Currently, the Optimize Table redistributes data at the granularity of partitions. If new data is ingest into a partition during the optimization process, conflicts can arise. The default strategy is to prioritize ingestion, causing the optimization job for that partition to fail. This makes it impossible for Optimize Table to handle:

• The latest partition of a partitioned table, which generally receives real-time updates.
• Non-partitioned tables, which are usually Primary Key tables and often receive real-time updates.

What I'm doing:

Support Online Optimize Table, allowing Optimize Table and DML operations to run concurrently without conflict.

Fixes #issue

What type of PR is this:

• [ ] BugFix
• [ ] Feature
• [x] Enhancement
• [ ] Refactor
• [ ] UT
• [ ] Doc
• [ ] Tool

Does this PR entail a change in behavior?

• [x] Yes, this PR will result in a change in behavior.
• [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
• [ ] Parameter changes: default values, similar parameters but with different default values
• [x] Policy changes: use new policy to replace old one, functionality automatically enabled
• [ ] Feature removed
• [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• [x] I have added test cases for my bug fix or my new feature
• [x] This pr needs user documentation (for new or modified features or behaviors)
  • [ ] I have added documentation for my new feature or new function
• [ ] This is a backport pr

Bugfix cherry-pick branch check:

• [x] I have checked the version labels which the pr will be auto-backported to the target branch
  • [x] 3.3
  • [ ] 3.2
  • [ ] 3.1
  • [ ] 3.0
  • [ ] 2.5

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
21.9% Duplication on New Code (required ≤ 3%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[github-actions[bot]]

[FE Incremental Coverage Report]

:white_check_mark: pass : 496 / 579 (85.66%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	com/starrocks/alter/OptimizeTask.java	0	6	00.00%	[54, 58, 59, 62, 66, 67]
:large_blue_circle:	com/starrocks/qe/StmtExecutor.java	7	22	31.82%	[2233, 2234, 2235, 2236, 2237, 2238, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2248, 2249]
:large_blue_circle:	com/starrocks/task/PublishVersionTask.java	5	7	71.43%	[78, 80]
:large_blue_circle:	com/starrocks/transaction/InsertTxnCommitAttachment.java	7	9	77.78%	[75, 76]
:large_blue_circle:	com/starrocks/catalog/OlapTable.java	11	13	84.62%	[383, 386]
:large_blue_circle:	com/starrocks/alter/OnlineOptimizeJobV2.java	356	407	87.47%	[135, 136, 143, 153, 171, 179, 204, 205, 206, 233, 245, 254, 328, 359, 360, 362, 363, 370, 371, 372, 376, 430, 439, 440, 441, 450, 451, 452, 454, 470, 486, 488, 495, 496, 497, 498, 511, 532, 536, 539, 540, 541, 547, 567, 596, 604, 627, 636, 759, 760, 770]
:large_blue_circle:	com/starrocks/transaction/DatabaseTransactionMgr.java	40	45	88.89%	[1039, 1040, 1201, 1202, 1203]
:large_blue_circle:	com/starrocks/transaction/OlapTableTxnStateListener.java	5	5	100.00%	[]
:large_blue_circle:	com/starrocks/catalog/Partition.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/planner/OlapScanNode.java	3	3	100.00%	[]
:large_blue_circle:	com/starrocks/common/Config.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/planner/OlapTableSink.java	25	25	100.00%	[]
:large_blue_circle:	com/starrocks/sql/ast/InsertStmt.java	4	4	100.00%	[]
:large_blue_circle:	com/starrocks/transaction/OlapTableTxnLogApplier.java	10	10	100.00%	[]
:large_blue_circle:	com/starrocks/alter/OptimizeJobV2Builder.java	9	9	100.00%	[]
:large_blue_circle:	com/starrocks/persist/gson/GsonUtils.java	1	1	100.00%	[]
:large_blue_circle:	com/starrocks/transaction/TransactionState.java	6	6	100.00%	[]
:large_blue_circle:	com/starrocks/transaction/PartitionCommitInfo.java	5	5	100.00%	[]

[github-actions[bot]]

[BE Incremental Coverage Report]

:white_check_mark: pass : 194 / 233 (83.26%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	be/src/runtime/load_channel_mgr.cpp	1	3	33.33%	[182, 183]
:large_blue_circle:	be/src/runtime/load_channel.cpp	14	20	70.00%	[161, 163, 164, 228, 230, 231]
:large_blue_circle:	be/src/storage/segment_replicate_executor.cpp	3	4	75.00%	[151]
:large_blue_circle:	be/src/agent/publish_version.cpp	15	20	75.00%	[169, 170, 171, 172, 173]
:large_blue_circle:	be/src/exec/multi_olap_table_sink.cpp	59	74	79.73%	[94, 95, 96, 98, 117, 118, 119, 120, 121, 123, 126, 129, 130, 131, 133]
:large_blue_circle:	be/src/storage/txn_manager.cpp	34	42	80.95%	[313, 314, 320, 321, 322, 323, 330, 348]
:large_blue_circle:	be/src/runtime/tablets_channel.h	6	7	85.71%	[109]
:large_blue_circle:	be/src/exec/tablet_sink.cpp	21	22	95.45%	[712]
:large_blue_circle:	be/src/runtime/lake_tablets_channel.cpp	1	1	100.00%	[]
:large_blue_circle:	be/src/exec/pipeline/fragment_executor.cpp	3	3	100.00%	[]
:large_blue_circle:	be/src/storage/tablet.cpp	8	8	100.00%	[]
:large_blue_circle:	be/src/exec/multi_olap_table_sink.h	1	1	100.00%	[]
:large_blue_circle:	be/src/exec/pipeline/olap_table_sink_operator.h	1	1	100.00%	[]
:large_blue_circle:	be/src/exec/tablet_sink.h	1	1	100.00%	[]
:large_blue_circle:	be/src/exec/tablet_sink_index_channel.cpp	3	3	100.00%	[]
:large_blue_circle:	be/src/exec/pipeline/olap_table_sink_operator.cpp	3	3	100.00%	[]
:large_blue_circle:	be/src/exec/data_sink.cpp	13	13	100.00%	[]
:large_blue_circle:	be/src/exec/async_data_sink.h	1	1	100.00%	[]
:large_blue_circle:	be/src/runtime/local_tablets_channel.cpp	3	3	100.00%	[]
:large_blue_circle:	be/src/storage/tablet_updates.cpp	3	3	100.00%	[]

[github-actions[bot]]

@Mergifyio backport branch-3.3

[mergify[bot]]

backport branch-3.3

✅ Backports have been created

• #47098 [Enhancement] Support online optimize table (backport #43747) has been created for branch branch-3.3 but encountered conflicts