flink icon indicating copy to clipboard operation
flink copied to clipboard
verified publisher icon apache

[FLINK-33472][Client] Solve the problem that the temporary file of flink-conf.yaml in S3AFileSystem cannot be uploaded

Open zhengzhili333 opened this issue 2 years ago • 5 comments

What is the purpose of the change

This pull request solve the problem that the temporary file of flink-conf.yaml in S3AFileSystem cannot be uploaded.

Brief change log

  • Modify YarnLocalResourceDescriptor# registerSingleLocalResource method's parameters, the parameters for the complete file path, the path contains scheme

Verifying this change

  • Added s3mock as a tool for unit testing s3 file systems
  • Added configuring fs.default-scheme as a prerequisite for triggering problems
  • Create temporary files and write content
  • Through YarnLocalResourceDescriptor# registerSingleLocalResource way to upload temporary files
  • Verify that the content of the uploaded temporary file is the same as that of the written temporary file

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

zhengzhili333 avatar Nov 23 '23 06:11 zhengzhili333

CI report:

  • 3ce2318aa823aeb682cbe71e5dfc5b75fc832ecb Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Nov 23 '23 07:11 flinkbot

@flinkbot run azure

zhengzhili333 avatar Nov 23 '23 08:11 zhengzhili333

Sorry for leaving you hanging for that long time. I was busy with other topics. Can you rebase the branch and bring the test case back? I hope I find time to review it in the next days.

XComp avatar Feb 21 '24 11:02 XComp

@XComp I have removed other commits and brought the test case back. Please review it when you have time.

zhengzhili333 avatar Feb 23 '24 06:02 zhengzhili333

@flinkbot run azure

zhengzhili333 avatar Feb 24 '24 03:02 zhengzhili333