connect
connect copied to clipboard
Add option for idempotent Kafka producer for Kafka output
Resolves #1223
Add a boolean option to kafka output to use idempotent producer
Thanks @abhishekamte, I think the description as it currently stands is a bit misleading, the guarantee of exactly-once delivery is only with respect to a single producer instance within the client library itself. In practice the majority of duplicates would be introduced during server crashes and catastrophic network failures as the services recover, in which case this field wouldn't help at all. I'm not 100% familiar with sarama internals but I'm assuming this based off https://stackoverflow.com/a/67909822 and the fact that we're not populating a transaction ID from the input.
I'd be happier with a more muted description such as: "Enable idempotent writes, this eliminates the possibility of message duplicates being written to a topic after recovering from network failures. Set this to true only if ack_replicas is true and Kafka version is above 0.11.0."
@Jeffail this is fairly important especially considering the default is true since Kafka 3.0.
@abhishekamte are you still around to make the changes? Or if not can I fork this PR and re-submit with the modifications requested by @Jeffail?