Highlevel consumer seek method takes more than 600ms for seeking end of the partition

[madanmch]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Do NOT create issues for questions, use the discussion forum: https://github.com/edenhill/librdkafka/discussions

Description

Kafka producer produces the messages every 30ms, but the consumer needs the latest messages for every 100ms. While consuming the messages, the consumer does "set_offset method to go OFFSET_END" and consume method to get the latest message. The consumer method takes more than 600ms most of the time. How can I make the process faster to read the message at end of the topic?

How to reproduce

A simple producer and consumer scenario, where the producer always produces more messages where consumers need the latest messages only.

I am using the below code to get the latest message from Kafka. Tried to use enable.auto.commit to false and auto.offset.reset to latest in conf & topic configurations, that doesn't help me.

std::vector<RdKafka::TopicPartition*> partitions;

partitions.push_back(RdKafka::TopicPartition::create(topic, 0, -1));

while (true){ consumer->seek(*partitions[0], 1000); RdKafka::Message *message = consumer->consume(1000); }

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] librdkafka version (release number or git tag): latest
• [x] Apache Kafka version: 3.0.0
• [x] librdkafka client configuration: mentioned above
• [x] Operating system: Ubuntu 18.04
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue

[madanmch]

@edenhill Can you please respond to this question? Thanks

[madanmch]

Can anyone respond to this question?

[edenhill]

Please fill out the checklist.

[madanmch]

Updated

[edenhill]

@madanmch the checklist explicitly asks you not say "latest" for the version, can you please provide either the released version number of a git sha? Thanks

Also reproduce this with debug=cgrp,topic,fetch configured and provide the logs.

Thanks