starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Test][Draft] DR test

Open sevev opened this issue 2 weeks ago • 2 comments

Why I'm doing:

What I'm doing:

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [x] Yes, this PR will result in a change in behavior.
  • [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [ ] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 4.0
    • [ ] 3.5
    • [ ] 3.4
    • [ ] 3.3

[!NOTE] Adds partition snapshot task and file sync pipelines with new RPCs/thread pools on BE and a FE job framework for automated cluster snapshots, plus related configs, metrics, and API updates.

  • Backend (BE):
    • Add PARTITION_SNAPSHOT task support: agent wiring, handler run_partition_snapshot_task, dedicated thread pool, shutdown handling, and metrics.
    • Implement lake partition snapshot execution (storage/lake/partition_snapshot_task*) that computes file diffs and invokes new RPC to upload files.
    • Add LakeService.upload_snapshot_files RPC and SnapshotFileSyncer to copy snapshot data files; create snapshot_file_syner thread pool in ExecEnv.
    • New config partition_snapshot_threads; expose corresponding metrics; add TBackend::operator<.
    • Enhance StarOS worker logging and FS building.
  • Frontend (FE):
    • Introduce snapshot job framework: ClusterSnapshotJob (abstract), FullClusterSnapshotJob, MetaClusterSnapshotJob, and SnapshotJobContext; refactor scheduler to use context and incremental state machine.
    • Wire finish handling for PARTITION_SNAPSHOT in LeaderImpl and ClusterSnapshotMgr.
    • Add ClusterSnapshotTask (thrift TPartitionSnapshotRequest) and batching in AgentBatchTask.
    • Extend StarOSAgent to allocate generic file paths and create virtual-tablet shard/group; persist vTablet info in StorageVolume.
    • Tests for cluster snapshot flow and full snapshot job.
  • IDL / Protocols:
    • Thrift: add TTaskType::PARTITION_SNAPSHOT, TPartitionSnapshotRequest, TPartitionSnapshotInfo and wire into agent/master APIs.
    • Proto: add UploadSnapshotFiles{Request,Response}PB and service RPC; define TabletDataSnapshotPB.

Written by Cursor Bugbot for commit 70034c6bcc1bf2eed5745edb785e63f5290ff297. This will update automatically on new commits. Configure here.

sevev avatar Dec 10 '25 03:12 sevev

🧪 CI Insights

Here's what we observed from your CI run for c3eb2219.

🟢 All jobs passed!

But CI Insights is watching 👀

mergify[bot] avatar Dec 10 '25 03:12 mergify[bot]

@cursor review

alvin-celerdata avatar Dec 10 '25 04:12 alvin-celerdata