flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-37422][state/forst] Respect the `maxTransferBytes` when using path copying in ForSt.

Open AlexYinHan opened this issue 6 months ago • 1 comments

What is the purpose of the change

This PR makes ForSt respect the maxTransferBytes when using path copying for snapshots, so that it does not upload oversized files.

Brief change log

  • In CopyDataTransferStrategy, use BytesCopy instead of PathCopy if the file size is greater than maxTransferBytes.

Verifying this change

This change added tests and can be verified as follows:

  • In DataTransferStrategyTest, a UT is added to test create snapshots with maxTransferBytes

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

AlexYinHan avatar Jun 19 '25 06:06 AlexYinHan

CI report:

  • 5df945a835652dabb8bffb57cd5e945a94e9eadf Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jun 19 '25 06:06 flinkbot