goka
goka copied to clipboard
lovoo
Feature: Exactly-once semantics
Is the exactly-once semantics added to the road map of the library? Would be very nice to have it. As it is the most wanted semantics in the stream processing realm.
Hi Nikolay, nope currently we don't have planned anything in this regard. If there's a bigger need for that however, we could at least check out the effort needed to add it to goka.
We definitely need it. It is a must for some applications where are some critical parts that need to ensure exactly-once delivery. And now it is easier than ever to implement: https://www.confluent.io/blog/enabling-exactly-once-kafka-streams/
We are using a Python faust which is a Stream Processing library with Exactly-Once Semantics available. And we want to switch to GoLang and the only thing which is stopping us is that there is no Streams Processing library with exactly-once semantics in GoLang.
Absolutely, I understand the necessity of this. It's simply a matter of time and ressources, so we have to evaluate how easy/fast it is to add this to the framework and if it's worth the effort. Let me check in the next days (or feel free to give it a look too), but I can't promise anything.
As a little bit of self promotion, my franz-go client has EOS support, if the problem for supporting EOS in goka is due to sarama's lack of EOS support.
It looks like sarama simply does not support transactions, therefore no EOS, and apparently no one is working on it.
Replacing the underlying library with something else like franz-go sounds like quite a major change, since goka is quite tightly coupled to sarama. Also I'd really like to give it a try, so thanks for the hint @twmb :) But for now, I'd say we simply cannot add support for EOS to goka in near/mid future with reasonable resources.
Currently, Sarama added transactional API from version v1.37.2, Here is a good article showing the implementation of transactions in producer/consumer - https://www.andreamedda.com/posts/play-with-sarama-transactional-api
