pranadb
pranadb copied to clipboard
Consider implementing custom Kafka consumer rebalance protocol
The Kafka rebalance protocol is not implemented in all Kafka clients (e.g. Segment IO).
The protocol is designed, such that, when a rebalance is in progress, consumer apps are given a chance to commit offsets before a partition is reassigned to another consumer. Without this, it's possible that duplicate messages could be delivered.
Instead of relying on clients to implement the rebalance protocol properly, we could implement a rebalance protocol ourselves in Prana and maintain our own mapping of consumers to partitions.