[QUESTION] extremly high Fetch Rate

[mosinnik]

I have http API service on fastapi that only write to kafka by aiokafka. So I only one producer cause very high fetch rate on monitoring graphs - usually 3.5-4k - that amount is sum of counts rates for different topics. Producer only send to one topic but fetch rate is also high for topics that are have no new messages for along time. For monitoring we use metric kafka_server_brokertopicmetrics_totalfetchrequests_total by topic and see that problem. Searching about metrics got https://newrelic.com/blog/how-to-relic/tuning-apache-kafka-consumers where found following text:

A reasonable fetch rate is somewhere around 1 to 5 requests per second per broker because it corresponds to 200 to 1,000 milliseconds of fetch latency. If your average fetch size is consistently hitting a limit and your average fetch request rate is higher than five requests per second per broker, you’re likely creating many small fetch requests instead of fewer large ones. This not only limits the throughput of your application, it can very quickly consume a large amount of your Kafka Broker capacity due to the high volume request rate which directly translates to significant costs.

As additional info: we also have java producers that don't generate parasitic fetches. Checked by stopping all kafka services one by one to find problem.

So few questions:

1. why aiokafka producer generate parasitic load?
2. why producer try fetch from topic even if it empty (we have some)?
3. is the way to disable this behavior to stop parasitic load on kafka?

[vmaurin]

For clarification, you are talking about "a producer", but fetch is called by consumers, so you have an issue with a consumer or a producer ?

[mosinnik]

yes, I have only one producer in this service, no consumers at all