Higher latency when all partitions are not receiving data

[vjrantal]

Description

When consuming via the Kafka API with a consumer that consumes multiple partitions, there is an additional latency introduced if data is not arriving to all consumed partitions.

How to reproduce

The repository at https://github.com/vjrantal/kafka-consumer-latency-test contains code to reproduce the behavior. Example output from the test code is available at https://github.com/vjrantal/kafka-consumer-latency-test/runs/2131371086?check_suite_focus=true#step:6:10 that shows the latencies from an example run.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed or where adequate information has not been provided.

Please provide the relevant information for the following items:

• [x] SDK (include version info): kafka-python 2.0.2 and same behavior is also reproduced with the dotnet sdk
• [x] Standalone repro https://github.com/vjrantal/kafka-consumer-latency-test

If this is a question on basic functionality, please verify the following:

• [x] Port 9093 should not be blocked by firewall ("broker cannot be found" errors)
• [x] Pinging FQDN should return cluster DNS resolution (e.g. $ ping namespace.servicebus.windows.net returns ~ ns-eh2-prod-am3-516.cloudapp.net [13.69.64.0])
• [x] Namespace should be either Standard or Dedicated tier, not Basic (TopicAuthorization errors)