kminion icon indicating copy to clipboard operation
kminion copied to clipboard

Cache topic info metric values

Open amuraru opened this issue 3 years ago • 3 comments

https://github.com/cloudhut/kminion/commit/77249c3b52f871625561ccfde07c2f1ae3626897 Introduced support for topic info metrics exported as labels. In large clusters with thousands of topics querying the topic configuration every scrape interval is an expensive operation leading to high scraping times and often to timeouts.

I am proposing we add support for caching the kafka topic config values and return from cache whenever the minion is scraped.

amuraru avatar Jul 28 '21 18:07 amuraru

Hey Adi, are you sure that the large scrape times are actually caused by the request that describes the topic configs? Did you measure it or how did you come to that conclusion?

If you use the adminApi as scrape mode my guess is that describing the consumer groups (offsets) takes the most time.

weeco avatar Jul 28 '21 19:07 weeco

Let me time this and get back - my observation so far is that on two different clusters with same number of consumer groups, the overall scrape time is 3x higher in the cluster with lot more topics(1200 vs 100)

amuraru avatar Jul 28 '21 19:07 amuraru

I also added https://github.com/cloudhut/kminion/issues/100 , as it's very cumbersome for you to debug this probably.

weeco avatar Jul 28 '21 19:07 weeco