Node Exporter 1.3.1 will break node metrics with prometheus-adapter is used

[lzecca78]

Hi! during a setup with module version 2.0.1 we've found out that node-exporter 1.3.1 have some issues when node metrics are requested (grafana dashboards that ask for them and kubectl top nodes). Once node-exporter is upgraded to 1.5.0 (at least), everything has back to work.

[nutellinoit]

Thank you @lzecca78 , we are in the process of releasing the new version of the module with node-exporter 1.5.0

[ralgozino]

I can't replicate the issue, could you give us some more details please?

This is what I did:

1. Created a 1.24 cluster with kind:

kind create cluster --image kindest/node:v1.24.7

2. Cloned monitoring v2.0.1

3. Deployed the following components without any patches:

   1. prometheus-operator
   2. prometheu-operated
   3. prometheus-adapter
   4. node-exporter
   5. kube-state-metrics
   6. kube-proxy-metrics
   7. kubeadm-sm

Wait a few minutes and kubectl top nodes should be working:

❯ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9", GitCommit:"9710807c82740b9799453677c977758becf0acbb", GitTreeState:"clean", BuildDate:"2022-12-08T10:15:09Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7", GitCommit:"e6f35974b08862a23e7f4aad8e5d7f7f2de26c15", GitTreeState:"clean", BuildDate:"2022-10-26T15:06:14Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
❯ kubectl top nodes
NAME                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
124-control-plane   247m         4%     2515Mi          31%
❯ kubectl describe pod -n monitoring node-exporter-x6kj2|grep -i 'image:'
    Image:         registry.sighup.io/fury/alpine:3
    Image:         registry.sighup.io/fury/prometheus/node-exporter:v1.3.1
    Image:         quay.io/brancz/kube-rbac-proxy:v0.12.0

In any case, the next release of the monitoring module will include node-exporter v1.5.0

[lzecca78]

Hi @ralgozino sorry for the delay! We have that issue with Kubernetes EKS 1.22. The answer that wer had from kubectl top nodes was metrics not available yet but forever. Once we've patched the version to 1.5.0 in a couple of minutes all the metrics were there.

[ralgozino]

fixed in #129