Suggestion: Secure alternative for scraping Kubernetes control-plane metrics (etcd, scheduler, controller-manager, proxy) using Prometheus Agent DaemonSet

[adrghph]

trafficstars

Is your feature request related to a problem ?

Many users encounter persistent issues when trying to scrape Kubernetes control-plane components using kube-prometheus-stack:

• Targets like etcd, kube-scheduler, kube-controller-manager, and kube-proxy often appear as DOWN
• Corresponding Services are usually ClusterIP: None or not accessible
• Static pod metrics are only exposed on 127.0.0.1, making them unreachable to Prometheus
• Workarounds require exposing components via 0.0.0.0, which is insecure :

#1704 #204

Describe the solution you'd like.

Instead of exposing control-plane components or modifying static pod manifests, i’ve built a secure alternative:

• A dedicated Helm chart that deploys a Prometheus Agent in DaemonSet mode on control-plane nodes
• It scrapes metrics locally on 127.0.0.1 (etcd, scheduler, controller-manager, kube-proxy)
• It pushes metrics to the central Prometheus instance via remote_write
• No additional Services or port exposure is required
• Works out-of-the-box with kubeadm defaults (no need to bind to 0.0.0.0)

Project link: https://github.com/adrghph/kps-zeroexposure

Describe alternatives you've considered.

1 - Modifying control-plane static pod manifests to bind metrics on 0.0.0.0 (not secure) 2 - Disabling metrics scraping for these components altogether (loses valuable observability)

Additional context.

This pattern follows the recommended architecture from the Prometheus team for remote_write agents.
It also aligns with security best practices by avoiding any form of service exposure or insecure binding.

This approach could be considered as an optional mode in the kube-prometheus-stack Helm chart (agentMode: true or similar).