flink
flink copied to clipboard
[FLINK-37422][state/forst] Respect the `maxTransferBytes` when using path copying in ForSt.
What is the purpose of the change
This PR makes ForSt respect the maxTransferBytes when using path copying for snapshots, so that it does not upload oversized files.
Brief change log
- In
CopyDataTransferStrategy, useBytesCopyinstead ofPathCopyif the file size is greater thanmaxTransferBytes.
Verifying this change
This change added tests and can be verified as follows:
- In
DataTransferStrategyTest, a UT is added to test create snapshots withmaxTransferBytes
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving): (no) - The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes)
- The S3 file system connector: (no)
Documentation
- Does this pull request introduce a new feature? (no)
CI report:
- 5df945a835652dabb8bffb57cd5e945a94e9eadf Azure: SUCCESS
Bot commands
The @flinkbot bot supports the following commands:@flinkbot run azurere-run the last Azure build