flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-35510][statebackend] Implement basic incremental checkpoint fo…

Open zoltar9264 opened this issue 9 months ago • 5 comments

…r ForStStateBackend

What is the purpose of the change

Use low DB api implement a basic incremental checkpoint for ForStStatebackend, follow steps:

  1. db.disableFileDeletions()
  2. db.getLiveFiles(true)
  3. db.entableFileDeletes(false)

Brief change log

  • Introduce ForStStateDataTransfer
  • Introduce ForStIncrementalSnapshotStrategy

Verifying this change

This change added tests and can be verified as follows:

  • Added test ForStStateDataTransferTest
  • Added test ForStIncrementalSnapshotStrategyTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

zoltar9264 avatar Jun 03 '24 13:06 zoltar9264