kafka-go Exactly once

Describe the solution you'd like Company I work for is using kafka as one of ways to communicate between services and GO is one of core languages we use.

We would love to use kafka-go, as it's API looks much nicer than Sarama, yet we have two requirements:

exactly once support for producer (so indepotence flag) suppoer
transactions support

With the latter I see #216 being in progress. How the situation looks with the former one?

We could contribute if help is needed, though we need to know current state before moving forward.

Supporting documentation https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Mar 15 '19 17:03 Jarema

Hi @Jarema ,

I'm not aware of any current work toward exactly once support. If your team is willing to tackle that part, any contribution is welcome !

Mar 15 '19 18:03 Pryz

Hi @Pryz. I will look into it. The problem is we have tight deadline on this and my decision depends on cost of implementation of at least exactly one feature.

In theory, it doesn't sound complicated:

each producer session has its own PID and a sequence number.
with each request, producer sends the PID and sequence
producer uses indepotence flag
if producer does not get ACK for a message, it resends it
if broker gets the same message as previous, but ACK was lost, it will send ACK duplicate message

So, with my limited kafka API knowledge and very limited kafka-go client knowledge it means that:

we need to add idepotence key when creating producer session
we need to iterate the seq key on each write
we need to check ACK duplicate response and treat it as a proper response (maybe current logic will handle that already if ACK is "similar enough".

Does it sound ok for you? If yes, I'll have a deeper look into kafka-go codebase to see how much work is needed to be done.

Mar 16 '19 07:03 Jarema

@Jarema Idempotence flag is also a requirement for supporting transactions. You can read it here: https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit# Look at pages 21-22. Please notify me about your PRs.