node_exporter
node_exporter copied to clipboard
Published
20 hours ago •
prometheus
prometheus
node-exporter daemonset keeps on creating new pods
Host operating system: output of uname -a
- Running as a pod on k8s cluster with ami type
AL2_x86_64andAL2_x86_64_GPU
node_exporter version: output of node_exporter --version
node-exportercontainer has the following image:quay.io/prometheus/node-exporter:v1.7.0
node_exporter command line flags
ARGSfrom the container
Args:
--path.procfs=/host/proc
--path.sysfs=/host/sys
--path.rootfs=/host/root
--path.udev.data=/host/root/run/udev/data
--web.listen-address=[$(HOST_IP)]:9100
node_exporter log output
Are you running node_exporter in Docker?
- No, its being run as a part of prometheus helm chart
v25.8.0
What did you do that produced an error?
- I simply created the
node-exporterwith default configuration, only adding tolerations of all of my nodegroups.
What did you expect to see?
- I expected to see one
node-exporterpod per node, and a pod only gets spun up in case a new node comes up.
What did you see instead?
- I see loads of
node-exporterpods being created on and on - daemonset output:
k get daemonset -n monitoring -o wide
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
prometheus-prometheus-node-exporter 12 12 12 12 12 kubernetes.io/os=linux 138m node-exporter quay.io/prometheus/node-exporter:v1.7.0 app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-node-exporter
k describe daemonset prometheus-prometheus-node-exporter -n monitoring
Name: prometheus-prometheus-node-exporter
Selector: app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-node-exporter
Node-Selector: kubernetes.io/os=linux
Labels: app.kubernetes.io/component=metrics
app.kubernetes.io/instance=prometheus
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=prometheus-node-exporter
app.kubernetes.io/part-of=prometheus-node-exporter
app.kubernetes.io/version=1.7.0
helm.sh/chart=prometheus-node-exporter-4.24.0
Annotations: deprecated.daemonset.template.generation: 1
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: monitoring
policies.kyverno.io/last-applied-patches:
autogen-tolerations-node-selectors.monitoring-tolerations-node-selectors.kyverno.io: added
/spec/template/spec/tolerations/2
Desired Number of Nodes Scheduled: 12
Current Number of Nodes Scheduled: 12
Number of Nodes Scheduled with Up-to-date Pods: 12
Number of Nodes Scheduled with Available Pods: 12
Number of Nodes Misscheduled: 0
Pods Status: 12 Running / 63 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/component=metrics
app.kubernetes.io/instance=prometheus
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=prometheus-node-exporter
app.kubernetes.io/part-of=prometheus-node-exporter
app.kubernetes.io/version=1.7.0
helm.sh/chart=prometheus-node-exporter-4.24.0
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
Service Account: prometheus-prometheus-node-exporter
Containers:
node-exporter:
Image: quay.io/prometheus/node-exporter:v1.7.0
Port: 9100/TCP
Host Port: 0/TCP
Args:
--path.procfs=/host/proc
--path.sysfs=/host/sys
--path.rootfs=/host/root
--path.udev.data=/host/root/run/udev/data
--web.listen-address=[$(HOST_IP)]:9100
Liveness: http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
HOST_IP: 0.0.0.0
Mounts:
/host/proc from proc (ro)
/host/root from root (ro)
/host/sys from sys (ro)
Volumes:
proc:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sys:
Type: HostPath (bare host directory volume)
Path: /sys
HostPathType:
root:
Type: HostPath (bare host directory volume)
Path: /
HostPathType:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 56m (x13 over 139m) daemonset-controller (combined from similar events): Created pod: prometheus-prometheus-node-exporter-jjkrl
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-twrkg
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-m5cc5
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-6t5ld
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-sdcb9
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-gmhlw
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-96b9z
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-98qnm
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-dwlbh
Normal SuccessfulCreate 56m daemonset-controller Created pod: prometheus-prometheus-node-exporter-5rjrt
- Events from pods in
Pendingstate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 57m (x2 over 57m) default-scheduler 0/13 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 12 node(s) didn't have free ports for the requested pod ports. preemption: 0/13 nodes are available: 1 Preemption is not helpful for scheduling, 12 No preemption victims found for incoming pod..
Warning FailedScheduling 51m (x4 over 57m) default-scheduler 0/13 nodes are available: 13 node(s) didn't have free ports for the requested pod ports. preemption: 0/13 nodes are available: 13 No preemption victims found for incoming pod..
Warning FailedScheduling 46m default-scheduler 0/13 nodes are available: 1 node(s) were unschedulable, 12 node(s) didn't have free ports for the requested pod ports. preemption: 0/13 nodes are available: 1 Preemption is not helpful for scheduling, 12 No preemption victims found for incoming pod..
Warning FailedScheduling 86s (x13 over 45m) default-scheduler 0/12 nodes are available: 12 node(s) didn't have free ports for the requested pod ports. preemption: 0/12 nodes are available: 12 No preemption victims found for incoming pod..
- All the
node-exporterpods in the namespace
k get pods -n monitoring -o wide | grep prometheus-prometheus-node-exporter
prometheus-prometheus-node-exporter-2tkt8 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-4hs56 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-57vxz 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-5m67m 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-5rjrt 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-68mlr 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-6mxvb 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-6t5ld 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-7qmtt 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-7t6wg 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-8f5hs 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-92qbc 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-96b9z 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-98qnm 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-9bh4p 1/1 Running 0 137m 10.8.136.64 ip-10-8-136-64.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-9jkmc 1/1 Running 0 137m 10.8.148.75 ip-10-8-148-75.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-9rw9s 1/1 Running 0 137m 10.8.161.180 ip-10-8-161-180.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-9svpj 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-bp6p8 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-c6nmg 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-c6ww9 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-ccjp5 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-cfqfj 1/1 Running 0 137m 10.8.154.220 ip-10-8-154-220.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-ckf58 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-cq664 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-d8tvf 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-dt95r 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-dwlbh 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-fx6xv 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-ggvvn 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-ghvbv 1/1 Running 0 137m 10.8.143.142 ip-10-8-143-142.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-gjls2 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-gmhlw 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-gxdzr 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-hx6q6 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-jg8g2 1/1 Running 0 137m 10.8.130.59 ip-10-8-130-59.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-jjkrl 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-jntnv 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-jr44s 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-jtvll 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-k2bc6 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-k58jl 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-kdd6d 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-kj9x2 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-llj5g 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-lzj22 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-m5cc5 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-m7rwp 1/1 Running 0 137m 10.8.139.85 ip-10-8-139-85.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-ms455 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-mxd5g 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-nwqrj 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-pfxf8 1/1 Running 0 137m 10.8.174.223 ip-10-8-174-223.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-qmmh6 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-rbh5r 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-rhf7v 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-rjq57 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-rmnth 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-rx6rz 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-s97fj 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-scgz9 1/1 Running 0 137m 10.8.153.113 ip-10-8-153-113.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-sdcb9 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-t4qnh 1/1 Running 0 137m 10.8.167.253 ip-10-8-167-253.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-t75lp 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-tdzjz 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-tp94n 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-tvrkc 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-tvw8t 1/1 Running 0 137m 10.8.156.134 ip-10-8-156-134.eu-west-1.compute.internal <none> <none>
prometheus-prometheus-node-exporter-twrkg 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-tzhls 0/1 Pending 0 54m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-vgr8p 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-vkdjb 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-vmqwf 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-wnlql 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-xjkcl 0/1 Pending 0 74m <none> <none> <none> <none>
prometheus-prometheus-node-exporter-ztdxt 1/1 Running 0 137m 10.8.172.16 ip-10-8-172-16.eu-west-1.compute.internal <none> <none>
