Throttling message consumption

[ksdvishnukumar]

Description

I have a C# application which uses the Confluent kafka package to read the message from Azure Eventhub. I have the below major setting from the Confluent Kafka Package. MaxPartitionFetchBytes =20971520, CancellationDelayMaxMs = 200 FetchMaxBytes = 5242880

With this major settings while consuming the messages i see the Throttle Request from Azure Eventhub. I have azure eventhub with Standard tier with 1TU, which means Egress: Up to 2 MB per second or 4096 events per second.

How to match this constrain in confluent kafka package setting.

But I don't see the message loss. but it takes time to consume the message.

How to control the throttling?

How to reproduce

Have Azure Eventhub Namespace of Standard tier with 1 TU, Create EH and 1 partition.

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [ ] Confluent.Kafka nuget version - 2.1.1
• [ ] Apache Kafka version
• [ ] Client configuration. CancellationDelayMaxMs = 200 MaxPartitionFetchBytes =20971520 and 1048579 FetchMaxBytes = 5242880 and 1048579
• [ ] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

@mhowlett @codyaray @ijuma @codyaray It will be helpful if any input on this

[ksdvishnukumar]

Couple of observation now after tuning the below configuration.

MaxPartitionFetchBytes to 1048576 (Default Value) FetchMaxBytes to 102400 message.max.bytes to 102399 CancellationDelayInMs to 10

With this I can see the Throttling Request happening in first 2 min when the application starts

[ksdvishnukumar]

Updating the question in this Comment

Problem Statement Single consumer is consuming the message from the single topic partition. The consuming topic partition has 8578 messages The observation from Azure Eventhub Metrics, Outgoing message is high and seeing the Throttling Request. This test is performed in Azure Eventhub with Standard Tier with 1 Throughput Unit.

Expected Result Should see only the 8578 message in Outgoing message metrics and no Throttling Request should see.

Confluent Kafka Nuget Package version 2.1.1 is used for this test

Consumer Configuration AutoOffsetReset =AutoOffsetReset.Earliest EnableAutoOffsetStore = true EnableAutoCommit = false AutoCommitIntervalMs = 10000 //Used only when "EnableAutoCommit " is set to true EnablePartitionEof = true ConnectionsMaxIdleMs = 180000 MaxPartitionFetchBytes = 12518400 // 11.9 MB PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky IsolationLevel = IsolationLevel.ReadUncommitted SocketNagleDisable = true SocketKeepaliveEnable = true MetadataMaxAgeMs = 180000 SessionTimeoutMs = 30000 MaxPollIntervalMs = 300000 CancellationDelayMaxMs = 200, Debug = "consumer,fetch"

Consumer Topic Metrics Note: The incoming messages is there is different time frame. Messages are accumulated once performing the test with different ConsumerGroup for each run for test

Producer Topic Metrics

Log details are attached in this

[ksdvishnukumar]

@mhowlett, @edenhill and Confluent Inc Members Any update on this?

[stylianosnicoletti]

any update on this ?