starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] Introduce dataVersion, versionEpoch and versionTxnType to partition

Open xiangguangyxg opened this issue 1 year ago • 3 comments

Why I'm doing:

  1. Help cross-cluster replication achieve eventual consistency in failover.
  2. Help cross-cluster replication support do not disable compaction when replicating to a shared-data cluster.

What I'm doing:

Introduce dataVersion, versionSeq and versionTxnType to partition:

  • dataVersion: dataVersion equals visibleVersion in shared-nothing mode, in shared-data mode, compactions increase visibleVersion but not dataVersion.
  • versionEpoch: versionEpoch uses to check consistency of partitions in source and target cluster.
  • versionTxnType: the type of the transaction to generate the version

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

xiangguangyxg avatar May 31 '24 09:05 xiangguangyxg

Quality Gate Failed Quality Gate failed

Failed conditions
5.0% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

sonarqubecloud[bot] avatar Jul 25 '24 11:07 sonarqubecloud[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 154 / 169 (91.12%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/leader/ReportHandler.java 0 1 00.00% [1742]
:large_blue_circle: com/starrocks/transaction/LakeTableTxnLogApplier.java 6 9 66.67% [54, 56, 91]
:large_blue_circle: com/starrocks/transaction/OlapTableTxnLogApplier.java 11 13 84.62% [181, 189]
:large_blue_circle: com/starrocks/catalog/Partition.java 32 37 86.49% [409, 410, 637, 640, 646]
:large_blue_circle: com/starrocks/transaction/TransactionType.java 13 15 86.67% [46, 59]
:large_blue_circle: com/starrocks/transaction/DatabaseTransactionMgr.java 16 18 88.89% [1277, 1299]
:large_blue_circle: com/starrocks/transaction/OlapTableTxnStateListener.java 1 1 100.00% []
:large_blue_circle: com/starrocks/replication/ReplicationJob.java 7 7 100.00% []
:large_blue_circle: com/starrocks/task/ClearTransactionTask.java 1 1 100.00% []
:large_blue_circle: com/starrocks/catalog/PhysicalPartitionImpl.java 32 32 100.00% []
:large_blue_circle: com/starrocks/catalog/system/information/PartitionsMetaSystemTable.java 3 3 100.00% []
:large_blue_circle: com/starrocks/replication/ReplicationTxnCommitAttachment.java 2 2 100.00% []
:large_blue_circle: com/starrocks/common/proc/PartitionsProcDir.java 14 14 100.00% []
:large_blue_circle: com/starrocks/transaction/LakeTableTxnStateListener.java 2 2 100.00% []
:large_blue_circle: com/starrocks/transaction/TransactionState.java 3 3 100.00% []
:large_blue_circle: com/starrocks/transaction/PartitionCommitInfo.java 6 6 100.00% []
:large_blue_circle: com/starrocks/task/PublishVersionTask.java 1 1 100.00% []
:large_blue_circle: com/starrocks/service/InformationSchemaDataSource.java 3 3 100.00% []
:large_blue_circle: com/starrocks/lake/TxnInfoHelper.java 1 1 100.00% []

github-actions[bot] avatar Jul 25 '24 14:07 github-actions[bot]

[BE Incremental Coverage Report]

:x: fail : 6 / 25 (24.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/exec/schema_scanner/schema_partitions_meta_scanner.cpp 6 25 24.00% [172, 174, 183, 189, 195, 201, 206, 211, 217, 228, 239, 244, 249, 255, 260, 265, 270, 275, 280]

github-actions[bot] avatar Jul 25 '24 14:07 github-actions[bot]

@Mergifyio backport branch-3.3

github-actions[bot] avatar Jul 26 '24 12:07 github-actions[bot]

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar Jul 26 '24 12:07 mergify[bot]