flink icon indicating copy to clipboard operation
flink copied to clipboard
verified publisher icon apache

[FLINK-28842][Connector/Kafka] Add client.id.prefix for the KafkaSink

Open sap1ens opened this issue 3 years ago • 3 comments

What is the purpose of the change

Adding a way to configure client.id.prefix for the KafkaSink, which, internally, configures client.id for the KafkaProducer. This implementation essentially copies the implementation of client.id.prefix in KafkaSource.

Brief change log

  • Added a new CLIENT_ID_PREFIX for the KafkaSink
  • Added a way to configure it for the KafkaSink
  • Updated KafkaWriter to use CLIENT_ID_PREFIX with the subtaskId to generate a client.id for the KafkaProducer
  • Also updated the documentation and added a test

Verifying this change

This change added a unit test and can be verified by running KafkaWriterITCase.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs

sap1ens avatar Aug 05 '22 21:08 sap1ens

CI report:

  • 1b194a7e7df0fdc97850792c3d9820e9d82eadae Azure: FAILURE
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Aug 05 '22 21:08 flinkbot

@flinkbot run azure

sap1ens avatar Aug 06 '22 02:08 sap1ens

@flinkbot run azure

sap1ens avatar Aug 06 '22 18:08 sap1ens

@PatrickRen Any thoughts?

MartijnVisser avatar Sep 26 '22 10:09 MartijnVisser

Thanks for the PR @sap1ens . As this is adding a new public API for Kafka sink, I think at least a discussion is required in the mailing list and we need vote for it. Do you mind starting a discussion thread in the dev and user mailing list?

PatrickRen avatar Oct 11 '22 09:10 PatrickRen

@PatrickRen thanks for reminding me about that, I kicked off the discussion.

sap1ens avatar Oct 12 '22 16:10 sap1ens

@sap1ens Do you want to move this PR to https://github.com/apache/flink-connector-kafka ?

MartijnVisser avatar Oct 11 '23 18:10 MartijnVisser