pykafka icon indicating copy to clipboard operation
pykafka copied to clipboard

why send "offset + 1" while sending (partitions, offset) to consumer.commit_offsets()

Open vijuSR opened this issue 6 years ago • 1 comments

For consumer.commit_offsets(partition_offsets), the documentation clearly mentions
"partition_offsets (Sequence of tuples of the form (pykafka.partition.Partition, int)) – (partition, offset) pairs to commit where partition is the partition for which to commit the offset and offset is the offset to commit for the partition.

But we need to send (partition, offset+1) to commit offset offset.

Also given, the default behavior is also "+1" type here

Wouldn't it make more sense if we can commit offset offset just by passing (partition, offset) and not (partition, offset+1)?

vijuSR avatar Apr 16 '19 11:04 vijuSR

The code is written this way to account for the fact that the returned offset from OffsetFetchRequest represents the offset of the next message to consume, rather than the last consumed message. OffsetCommitRequest also expects the offset of the next message to be consumed. See https://github.com/Parsely/pykafka/issues/289 for details. It would probably be clearer to write this logic in terms of next_offset instead of last_offset_consumed in the SimpleConsumer.

emmettbutler avatar Apr 17 '19 18:04 emmettbutler