Prometheus additional scrape config scrapes entire cluster

[F-Plesa]

Discussed in https://github.com/strimzi/strimzi-kafka-operator/discussions/7095

^{Originally posted by F-Plesa July 22, 2022} Hello, we are deploying Strimzi on OpenShift (with seperate Prometheus instance, not using openshift-monitoring) and have a problem with additionalScrapeConfig file provided in example/metrics. As it is, it uses role: node. There is a namespaces: names: [] field but since nodes are not namespaced object, Prometheus (in my experience) disregards that field entirely.

The result of this is Prometheus scraping nodes for additional metrics such as CPU usage of Kafka brokers which is nicely shown in Grafana. However, Prometheus is also scraping CPU usage for all other deployments in all other namespaces. This is a big issue for two reasons: performance and security. In our case we have separate namespaces "kafka" and "monitoring" and we want to monitor Kafka deployment in "kafka" namespace. But, when exploring metrics in Prometheus I see metrics from all other namespaces as well.

Since role: node cannot be namespaced (if it can, please correct me), is it possible to implement the resource usage metrics with role: pod instead so it can be namespaced? The approach of "scrape everything and display what you need" is a problem, as described, especially in big clusters. Realistically what should be metrics for 6 pods (3 brokers, 3 zookeepers) ends up being thousands of metrics from all over the place.

Not sure if this should be filed under Discussions or Issues so I am posting it here for now.

Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config Strimzi additional scrape config: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-additional-properties/prometheus-additional.yaml

[scholzj]

I don't really understand why are you opening an issue out of the discussion. As explained there, the Prometheus files are just examples of how you can configure it. If you wanna do it differently, you can configure your Prometheus instance in whatever way you want.

[scholzj]

Triaged 25.8.2022: Should be closed. We can continue the discussion under #7095 if there is some actual bug to address and then open and issue about the bug.