flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-32087][checkpoint] Introduce space amplification statistics of file merging

Open fredia opened this issue 1 year ago • 1 comments

What is the purpose of the change

This PR introduces space amplification statistics of file merging.

Brief change log

  • Add SpaceStat class in FileMergingSnapshotManager.
  • Update SpaceStat on physical/logical files's creation/deletion

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

  • Add FileMergingSnapshotManagerTestBase#testSpaceStat
  • Update FileMergingSnapshotManagerTestBase#testRestore
  • Update testCreateAndReuseFiles/testCheckpointNotification

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

fredia avatar May 08 '24 04:05 fredia

CI report:

  • 6aa55b61e4440c4380b133a4db503bf3a13d75e1 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar May 08 '24 04:05 flinkbot

Thanks for the update! Overall LGTM. And will this SpaceStat report to metrics?

Currently no, will report to metric in FLINK-32091.

fredia avatar May 14 '24 03:05 fredia