[prometheus-node-exporter] Use PodMonitor instead of ServiceMonitor

[TaylorMutch]

Is your feature request related to a problem ?

Since node-exporter is used for node-level metrics, there shouldn't really be a need to create a Service and ServiceMonitor for the pods. We can skip creating a Service which allocates a ClusterIP, and instead use a PodMonitor. Additionally, it will allow us to leverage node metadata (see this PR and issue), which will only be supported by Prometheus pod role and not endpoints role.

Describe the solution you'd like.

Switch to using a PodMonitor for this chart instead of a ServiceMonitor.

Describe alternatives you've considered.

None

Additional context.

No response

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[bootc]

This would be really handy, as it's currently quite complicated to cross-reference metrics from e.g. the kubelet exporter and the node exporter (you have to play games stripping off port numbers, etc).

[paulfantom]

One thing to keep in mind is that node_exporter is using hostNetwork and thus it is exposed outside the cluster. kube-prometheus (jsonnet one, not helm) deals with this problem by injecting a kube-rbac-proxy sidecar container to have authentication options. However the downside of this solution is that the prometheus needs access to bearerTokenFile to authenticate when scraping data. Since PodMonitor is not supporting bearerTokenFile, the project also won't be moving away from ServiceMonitor. But that's from the perspective of kube-prometheus :)

Note that there is an upstream issue - https://github.com/prometheus-operator/prometheus-operator/issues/4523 that tries to deal with this limitation of PodMonitor. When it is finished, kube-prometheus will most likely start investigation into switching from ServiceMonitor to PodMonitor. I imagine kube-prometheus-stack following afterwards :)

Basically what I mean is that there is a huge security trade-off for using PodMonitor for node_exporter.

[TaylorMutch]

@paulfantom thanks so much for the info! Perhaps it's best to wait until that issue is resolved, then we can switch to PodMonitor here.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.