TLS is disabled for node_exporter

[jsturtevant]

What happened? I was looking through the specs generated and the found the message:

kubectl logs node-exporter-cbgwz  
evel=info ts=2021-11-04T22:21:56.153Z caller=tls_config.go:191 msg="TLS is disabled." http2=false

even though secure-listen-address is set https://github.com/prometheus-operator/kube-prometheus/blob/31d86b794645315e8a0c270f7eed1d0231a68e29/manifests/node-exporter-daemonset.yaml#L57

from what I can tell you have to use --web.config.file to get tls: https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md and https://github.com/prometheus/node_exporter/pull/1277 and I wasn't able to find the secure-listen-address in the node-exporter code.

Did you expect to see some different?

How to reproduce it (as minimally and precisely as possible): deploy the manifests from main branch and look at logs of node_exporter pod

Environment k8s

• Prometheus Operator version:

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?: related to https://github.com/prometheus-operator/kube-prometheus/issues/469 and https://github.com/prometheus-operator/kube-prometheus/issues/179

[paulfantom]

kube-prometheus is deploying node-exporter with kube-rbac-proxy sidecar. Sidecar is responsible for RBAC and TLS termination.

We are using this architecture for historical reasons as node_exporter did have TLS termination capabilities before. It might be good to consider updating to use TLS termination in node_exporter itself.

[jsturtevant]

Thanks for the clarification. If either of the other issues linked covers this feel free to close.

[raptorsun]

If we move TLS termination to node-exporter, can kube-rbac-proxy passthrough the TLS connection to upstream node-exporter? I'm thinking to add an option to move TLS termination to node-exporter while kube-rbac-proxy takes care of access control only.

[paulfantom]

@raptorsun kube-rbac-proxy realizes 2 functions: TLS termination and access restriction. As such moving TLS termination to node_exporter won't allow us to remove kube-rbac-proxy sidecar.

Additionally enabling TLS termination in node_exporter means something would need to provision those certificates. Maybe cert-manager as in https://github.com/prometheus-operator/kube-prometheus/issues/179?

[raptorsun]

What is the feature required from this issue? Is it an option to expose HTTPS port of node-exporter and disable kube-rbac-proxy? Excuse me plz but I am a little bit confused 😅

[paulfantom]

The issue is about removing kube-rbac-proxy, using node_exporter buillt-in mechanisms for TLS termination and auth. It needs a bit of exploratory work to determine if this is even possible.

[raptorsun]

ok, I will check its feasability first and see if we can proceed to add this feature.

[raptorsun]

Feasibility check result: Node Exporter is able to terminate TLS and authenticate client by its certificate. It can let pass the client certificates issued by a certain CA while blocking the clients with certificates issued by other CAs. However, it is not capable to determine the role of a client as kube-rbac-proxy does.

The diagram below shows the maximum we can do without kube-rbac-proxy.

This diagram shows how kube-rbac-proxy is used to verify the role of a client with a certificate.

[philipgough]

Just to point out, TLS in node_exporter is marked as experimental as per https://github.com/prometheus/node_exporter#tls-endpoint