pykafka
pykafka copied to clipboard
why send "offset + 1" while sending (partitions, offset) to consumer.commit_offsets()
For consumer.commit_offsets(partition_offsets), the documentation clearly mentions
"partition_offsets (Sequence of tuples of the form (pykafka.partition.Partition, int)) – (partition, offset) pairs to commit where partition is the partition for which to commit the offset and offset is the offset to commit for the partition.
But we need to send (partition, offset+1) to commit offset offset.
Also given, the default behavior is also "+1" type here
Wouldn't it make more sense if we can commit offset offset just by passing (partition, offset) and not (partition, offset+1)?
The code is written this way to account for the fact that the returned offset from OffsetFetchRequest represents the offset of the next message to consume, rather than the last consumed message. OffsetCommitRequest also expects the offset of the next message to be consumed. See https://github.com/Parsely/pykafka/issues/289 for details. It would probably be clearer to write this logic in terms of next_offset instead of last_offset_consumed in the SimpleConsumer.