Need to understand delay in fetching message using ReadMessage

[jforjava1981]

Description

I have set up a local kafka cluster with single topic having 3 partitions. I have written a standalone golang program using this API to seek and read message from particular offset and partition. I am trying to seek with offset and partition number I already have for 100 messages. I follow the call to seek with readMessage(). Most of the times readMessage returns in microseconds. Howeve for every few transactions it takes around 500 milliseconds. Can you put some light on why this could be happning and if there is any way to reduce the time or number of such occurences. e..g for 100 messages read with seek following by readMessage 13 message take around 500 milliseconds while 88 of them only take microseconds. I know maybe this is nothing to do with the API. However Can you put some light on why I get such a behaviour?

How to reproduce

Set up a local kafka cluster with single topic and 3 partitions. Create a simple golang program to first seek a mesasge for a particular offset and partition and then call ReadMessage to get the message at that particular offset and partition. Set the auto commit off for the consumer. log the time before readMessage API call and after the readMessage API call. You will observe that most of the messages take microseconds. However periodically some take 500 milliseconds.

Checklist

Please provide the following information:

• [x] confluent-kafka-go and librdkafka version (LibraryVersion()): 1.9.2
• [x] Apache Kafka broker version: kafka_2.13-3.0.0
• [x] Client configuration: ConfigMap{...} kafkaConfigMap := kafka.ConfigMap{ "bootstrap.servers": p.Properties.Bootstrapper, "group.id": Indexer, "partition.assignment.strategy": "cooperative-sticky", "auto.offset.reset": "smallest", "enable.auto.commit": false, }
• [x] Operating system: WSL2 ubuntu on windows 10
• [x] Provide client logs (with "debug": ".." as necessary)
• [x] Provide broker log excerpts
• [] Critical issue

[jforjava1981]

@mhowlett @edenhill can you please help?