Question - Kafka Producer performance issue question when Kafka consumer also acts as producer to another Kafka topic

[Sandesh-J]

Description

We have developed a .NET CORE service. The service consumes a message from a topic and produces a message to another topic. So the same service is acting as consumer for one topic and producer for another topic. When the service is publishing a message to kafka topic, then it is taking very long time(around 10 seconds per message on an average) to publish the message. We have used linger.ms settings and increased it value to 100 from default 0 value. So it has improved the performance a bit. But still the performance had not improved at a acceptable level to us. It is still taking around 5 seconds per message on an average to publish message. The service is using Schema Registry when publishing the message. we are using v1.2.0 librdkafka at this moment. (Confluent.Kafka nuget version. is 1.2.0 and Confluent.SchemaRegistry 1.2.0) https://github.com/confluentinc/librdkafka/blob/v1.2.0/CONFIGURATION.md

Below is the line of code which is taking good amount of time to publish a message

DeliveryResult<TKey, TValue> result = await _producer.ProduceAsync("TopicNameHere", messageToSend).ConfigureAwait(continueOnCapturedContext: false);

Right now below details are used when creating producer configuration with linger.ms to 100. We have also tried changing linger.ms value to 200, 500, 750, 1000. Increasing its value beyond 100 does not give us any additional benefit in the performance.

bootstrap.servers
linger.ms

So I have below questions with respect to the performance issue we are facing while publishing a message to Kafka topic

1. Any thoughts on what additional things we can do to improve the performance of producer?
2. Are we using Kafka correctly when same service acts as consumer and producer at the same time? In this case , for each message consumed from one topic, the service has to publish a message to another topic.

How to reproduce

Details mentioned in the description

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version. : It is 1.2.0
• [ ] Apache Kafka version.
• [ ] Client configuration.
• [ ] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[drawxy]

I have encountered the same issue. The version of confluent client is 2.3.0 and the .Net is 6.0. Is there any solution to improve the producing latency? Thanks!

[ksdvishnukumar]

I do also encountered the same behavior..Not aire how to solve this.

If any one came across and fixed kindly share the details helps for the users..

[betmix-matt]

We had this same problem and it was because we were creating the Producer with each message we were producing instead of reusing the producer. You'll have to provide more code to know if that's what you're doing as well. Producing a message to Kafka should take less than 50 milliseconds in general even with 2 in sync replicas.

[Sandesh-J]

In my case, I was creating a single producer for all the messages, still I was getting the issue. The service was using schema registry as well. After trying all the combinations of configurations, only solution that fixed the issue in my case was scaling out the producer service to have multiple instances. After multiple instances of the producer service(The service which produces the messages) were created, the issue got resolved. My guess is - with single instance of producer service, it was hitting some network/connection limitations which appears to be resolved by scaling out the service by creating few additional instances.