flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-28773][hive] Fix Hive sink not write a success file after finish writing in batch mode

Open Tartarus0zm opened this issue 3 years ago • 2 comments

What is the purpose of the change

Allow to write a success file after finish for Hive sink in batch mode

Brief change log

Allow PartitionLoader to use SuccessFileCommitPolicy to create success file when commit partition.

Verifying this change

add UT

  • FileSystemCommitterTest#testWriteSuccessFile

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (not documented)

Tartarus0zm avatar Aug 05 '22 09:08 Tartarus0zm

CI report:

  • 7b2f1fb6da00ce268ad28f47ca2fa4ac3822945f Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Aug 05 '22 09:08 flinkbot

@wuchong thanks for your review! I'll update this pr as soon

Tartarus0zm avatar Aug 08 '22 11:08 Tartarus0zm

Hi @Tartarus0zm , do you have time to update the PR?

wuchong avatar Aug 15 '22 03:08 wuchong

I appended a commit to address the comments. Will merge it if you don't have other concerns and CI is passed.

@wuchong I appended a commit to fix the case where the partitionCommitPolicyFactory is null in the FileSystemOutputFormat

Tartarus0zm avatar Aug 15 '22 16:08 Tartarus0zm