flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-33986][runtime] Extend ShuffleMaster to support snapshot and restore state.

Open JunRuiLee opened this issue 9 months ago • 1 comments

What is the purpose of the change

Extend ShuffleMaster to support snapshot and restore state.

Brief change log

Extend shuffleMaster to support batch snapshot as follows:

  1. Add method supportsBatchSnapshot to identify whether the shuffle master supports taking snapshot in batch scenarios
  2. Add method snapshotState and restoreState to snapshot and restore the shuffle master's state.

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

JunRuiLee avatar May 13 '24 01:05 JunRuiLee

CI report:

  • 0a268a5463696e20e06e05b5bd6355d8663455f8 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar May 13 '24 01:05 flinkbot

Thanks @zhuzhurk for review, I've updated this pr accordingly. PTAL~

JunRuiLee avatar May 14 '24 08:05 JunRuiLee