confluent-kafka-python icon indicating copy to clipboard operation
confluent-kafka-python copied to clipboard

How do I get the the offset of last message of a Kafka topic

Open tigerinus opened this issue 3 years ago • 1 comments

Description

I need to retrive last N messages of a topic using confluent-kafka-python.

I've been reading https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html# for a day, but no finding any appropriate method for getting the offset of the last message, thus I cannot calculate the offset for consumer to start with.

Please help. Thanks!

version: 1.8.2
libversion: 1.6.2
broker version: 3.0.0
client configuration:  just {'bootstrap.servers': boot_strap_servers}
os: linux

Checklist

Please provide the following information:

  • [x] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
  • [x] Apache Kafka broker version:
  • [x] Client configuration: {...}
  • [x] Operating system:
  • [ ] Provide client logs (with 'debug': '..' as necessary)
  • [ ] Provide broker log excerpts
  • [ ] Critical issue

tigerinus avatar Feb 19 '22 23:02 tigerinus

Hi @tigerinus , thanks for asking. This is the method to retrieve low and high offsets for the specified partition: https://github.com/confluentinc/confluent-kafka-python/blob/master/src/confluent_kafka/src/Consumer.c#L1375

jliunyu avatar Feb 22 '22 06:02 jliunyu