connect icon indicating copy to clipboard operation
connect copied to clipboard

Add option for idempotent Kafka producer for Kafka output

Open abhishekamte opened this issue 2 years ago • 2 comments

Resolves #1223

Add a boolean option to kafka output to use idempotent producer

abhishekamte avatar Jun 03 '22 21:06 abhishekamte

Thanks @abhishekamte, I think the description as it currently stands is a bit misleading, the guarantee of exactly-once delivery is only with respect to a single producer instance within the client library itself. In practice the majority of duplicates would be introduced during server crashes and catastrophic network failures as the services recover, in which case this field wouldn't help at all. I'm not 100% familiar with sarama internals but I'm assuming this based off https://stackoverflow.com/a/67909822 and the fact that we're not populating a transaction ID from the input.

I'd be happier with a more muted description such as: "Enable idempotent writes, this eliminates the possibility of message duplicates being written to a topic after recovering from network failures. Set this to true only if ack_replicas is true and Kafka version is above 0.11.0."

Jeffail avatar Jun 18 '22 08:06 Jeffail

@Jeffail this is fairly important especially considering the default is true since Kafka 3.0.

@abhishekamte are you still around to make the changes? Or if not can I fork this PR and re-submit with the modifications requested by @Jeffail?

rupurt avatar Oct 12 '22 08:10 rupurt