fluent-plugin-kafka icon indicating copy to clipboard operation
fluent-plugin-kafka copied to clipboard

Published 20 hours ago • verified publisher icon fluent

out_rdkafka2: enable idempotency for kafka producers

Open dipendra-singh opened this issue 1 year ago • 4 comments

Which issue(s) this PR fixes: None.

What this PR does / why we need it: In order to enable Idempotency for kafka producers, rdkafka allowes us to enable this. (here)

Docs Changes: README.md : Added to usage template for better visibility.

Release Note: Same as the title.

dipendra-singh avatar Dec 01 '23 18:12 dipendra-singh

Any rdkafka options can be set by rdkafka_options parameter. e.g.)

<match tag>
  @type rdkafka2
   rdkafka_options {                                                                                                                                        
     "enable.idempotence" : true
   }
   ....
</match>

Is there any reason to add a new specific parameter for idempotence?

ashie avatar Dec 06 '23 08:12 ashie

@ashie To keep the consistency between this and other plugins. One needs to explore rdkafka options to have similar features to that of other plugins.

dipendra-singh avatar Dec 06 '23 09:12 dipendra-singh

Ah, I see. Thanks for clarification. It seems reasonable.

https://github.com/fluent/fluent-plugin-kafka/blob/ea0f10a711a3cb49fba2a2c36dd3c3efa5b758ad/lib/fluent/plugin/out_kafka2.rb#L37

ashie avatar Dec 06 '23 09:12 ashie

hey @ashie Can we close this?

dipendra-singh avatar Dec 11 '23 06:12 dipendra-singh

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days

github-actions[bot] avatar Jul 30 '24 10:07 github-actions[bot]

Thanks for your contribution! I'm sorry for delay merging this.

ashie avatar Jul 31 '24 05:07 ashie