flink icon indicating copy to clipboard operation
flink copied to clipboard
verified publisher icon apache

[FLINK-32075][FLIP-306][Checkpoint] Delete merged files on checkpoint abort or subsumption

Open Zakelly opened this issue 1 year ago • 7 comments

What is the purpose of the change

As important part of the FLIP-306, this PR enables the file deletion on checkpoint abort or subsumption.

Brief change log

  • Track all logic files
  • Wire checkpoint notification with FileMergingSnapshotManager

Verifying this change

  • Added test in FileMergingSnapshotManagerTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Zakelly avatar Jan 23 '24 16:01 Zakelly

CI report:

  • 99ccd8ceababe4a80df6ccdb3074e2d7dfdf0a30 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jan 23 '24 16:01 flinkbot

@fredia @masteryhx Would you please take a look? Thanks!

Zakelly avatar Jan 24 '24 02:01 Zakelly

@fredia Thanks for your suggestions! I have addressed them, please take another look! Many thanks!

Zakelly avatar Feb 04 '24 16:02 Zakelly

The CI failure is not related with this PR since I only add comments based on last CI green.

Zakelly avatar Feb 18 '24 09:02 Zakelly

@flinkbot run azure

Zakelly avatar Feb 18 '24 10:02 Zakelly

@flinkbot run azure

Zakelly avatar Feb 18 '24 14:02 Zakelly

@flinkbot run azure

Zakelly avatar Feb 19 '24 02:02 Zakelly

Rebased master and addressed comments. Force pushed. @masteryhx would you please take another look? Thanks!

Zakelly avatar Feb 29 '24 10:02 Zakelly