kminion icon indicating copy to clipboard operation
kminion copied to clipboard

Add fine control on what consumer groups metrics are exposed to reduce scraped metrics in large clusters

Open amuraru opened this issue 3 years ago • 0 comments

For large clusters with tens of thousands of consumer groups, the number of metrics exposed may be quite large. This in turn poses extra pressure on scraping Prometheus as all these series names need to kept in RAM.

Few suggestions:

  • add flag to disable exposing metrics for inactive consumer groups (Empty or Dead)
  • List offsets for Stable consumer groups only : reduce the number of metrics for clusters where lots are consumer groups are short-lived and empty.
  • Report consumer_group_empty_members metrics for stable groups only

See https://github.com/cloudhut/kminion/pull/105#discussion_r680732016

amuraru avatar Aug 03 '21 08:08 amuraru