flink
flink copied to clipboard
[FLINK-32087][checkpoint] Introduce space amplification statistics of file merging
What is the purpose of the change
This PR introduces space amplification statistics of file merging.
Brief change log
- Add
SpaceStatclass inFileMergingSnapshotManager. - Update
SpaceStaton physical/logical files's creation/deletion
Verifying this change
Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.
This change added tests and can be verified as follows:
- Add
FileMergingSnapshotManagerTestBase#testSpaceStat - Update
FileMergingSnapshotManagerTestBase#testRestore - Update
testCreateAndReuseFiles/testCheckpointNotification
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving): (no) - The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
CI report:
- 6aa55b61e4440c4380b133a4db503bf3a13d75e1 Azure: SUCCESS
Bot commands
The @flinkbot bot supports the following commands:@flinkbot run azurere-run the last Azure build
Thanks for the update! Overall LGTM. And will this
SpaceStatreport to metrics?
Currently no, will report to metric in FLINK-32091.