pranadb icon indicating copy to clipboard operation
pranadb copied to clipboard

Consider implementing custom Kafka consumer rebalance protocol

Open purplefox opened this issue 3 years ago • 0 comments

The Kafka rebalance protocol is not implemented in all Kafka clients (e.g. Segment IO).

The protocol is designed, such that, when a rebalance is in progress, consumer apps are given a chance to commit offsets before a partition is reassigned to another consumer. Without this, it's possible that duplicate messages could be delivered.

Instead of relying on clients to implement the rebalance protocol properly, we could implement a rebalance protocol ourselves in Prana and maintain our own mapping of consumers to partitions.

purplefox avatar Sep 30 '21 10:09 purplefox