confluent-kafka-dotnet
confluent-kafka-dotnet copied to clipboard
confluentinc
Slow consumer on some topic/partitions
Description
We have several consumers instances running inside same application. Each consumer subscribed to different topic, each topic with different message type. Topics have very different partition numbers, ex: 1 topic 30partitions, another topic 1 or 2 partitions.
We have noticed that the topic with few partitions number (1 or 2) is consumed very slowly compared with the topic with large partitions number. Volume of messages and size is bigger in topic with 30 partitions (thousands). In topic with 1 or 2 partitions there is less than hundreds messages.
Any explanation of this situation ?
There is any way to define priority for a consumer instance or at topic level ??
Description
We have several consumers instances running inside same application. Each consumer subscribed to different topic, each topic with different message type. Topics have very different partition numbers, ex: 1 topic 30partitions, another topic 1 or 2 partitions.
We have noticed that the topic with few partitions number (1 or 2) is consumed very slowly compared with the topic with large partitions number. Volume of messages and size is bigger in topic with 30 partitions (thousands). In topic with 1 or 2 partitions there is less than hundreds messages.
Any explanation of this situation ?
There is any way to define priority for a consumer instance or at topic level ??
Hi @alfhv actually your issue is a bit hard to understand ! because it could be because of your Kafka hasn't configured well on the server etc.. , anyway you can define queue priority. for more info please check link, click on link
this priority configuration exists in dotnet implementation ??
based on my knowledge , you cannot define priority on C#/.Net
There is any way to define priority for a consumer instance or at topic level ??
no. the logic for determining which partitions to fetch is fixed.
i can see some benefit in making this an advanced configuration (though it is unlikely to get prioritized soon).
We have several consumers instances running inside same application.
oh different consumer instances? everything is independent. This seems more likely something broker side to me. Inspecting the debug logs may give you some insight as where the difference is coming from. you might also try fiddling the configuration to try and impact the relative buffering in each case. you could maybe blame fetch.min.bytes except this has a default value 1.
i'm closing this because i'm going through cleaning up the issues and expect that on the balance of probabilities this doesn't reflect an issue with the client.