reactor-kafka icon indicating copy to clipboard operation
reactor-kafka copied to clipboard

Periodic committer does not commit already committed offsets

Open bsideup opened this issue 6 years ago • 0 comments

In Kafka, the offsets are stored for a limited period of time (24h by default), and if the consumer did not commit them during that period, they get wiped. There is "periodic commit" logic in reactor-kafka, but it removes consumedOffsets during the commit: https://github.com/reactor/reactor-kafka/blob/3bd196219f7c7262725b94355e31abce7592b79b/src/main/java/reactor/kafka/receiver/internals/CommittableBatch.java#L67

If partition did not receive any events for the past 24h, no commit will be performed and Kafka will delete the offset for that partition.

Reactor Kafka should support a "refresh" of already committed offsets to avoid that.

See a similar issue in Akka's reactive-kafka: https://github.com/akka/reactive-kafka/pull/375

bsideup avatar Feb 17 '18 21:02 bsideup