kminion
kminion copied to clipboard
Cache topic info metric values
https://github.com/cloudhut/kminion/commit/77249c3b52f871625561ccfde07c2f1ae3626897 Introduced support for topic info metrics exported as labels. In large clusters with thousands of topics querying the topic configuration every scrape interval is an expensive operation leading to high scraping times and often to timeouts.
I am proposing we add support for caching the kafka topic config values and return from cache whenever the minion is scraped.
Hey Adi, are you sure that the large scrape times are actually caused by the request that describes the topic configs? Did you measure it or how did you come to that conclusion?
If you use the adminApi as scrape mode my guess is that describing the consumer groups (offsets) takes the most time.
Let me time this and get back - my observation so far is that on two different clusters with same number of consumer groups, the overall scrape time is 3x higher in the cluster with lot more topics(1200 vs 100)
I also added https://github.com/cloudhut/kminion/issues/100 , as it's very cumbersome for you to debug this probably.