node_exporter

node_exporter copied to clipboard

Host operating system: output of uname -a

Linux scnomgmt08 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
  build user:       root@586879db11e5
  build date:       20230717-12:10:52
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

Including commented out flags that I've tried that also had no success

- --log.level=debug
- --collector.netstat.fields=.*
- --collector.filesystem.mount-points-exclude=^(.*)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
#- --no-collector.filesystem
#- --no-collector.diskstats
#- --no-collector.filefd
#- --no-collector.mountstats
#- --no-collector.xfs
#- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|var/lib/containerd/.+|host/root/.+|host/.+)($|/)
# copied from Anthos
#- --collector.filesystem.mount-points-exclude=^/(host/root/|host/)?(dev|proc|sys|var/lib/docker/.+)($|/)
#- --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$

node_exporter log output

ts=2023-09-06T22:48:07.504Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/kubelet/pods/385f109c-132c-4d61-bd6f-bdcc0ba345cf/volumes/kubernetes.io~portworx-volume/pvc-ca28e648-3a66-4bc9-ba8c-3d5d9b313406

Are you running node_exporter in Docker?

Kubernetes with containerd

What did you do that produced an error?

Pod with PVC attached is deleted  and gets stuck in terminating.  It will hang here forever until force deleted, node exporter cycled, or manual filesystem unmount

What did you expect to see?

Node exporter ignores the PVC filepath on the host system, according to the regex given.  

What did you see instead?

It appears that node exporter does not ignore the filepaths correctly, because it maintains an OPENFILE on the pvc mount path. Output from a custom script written to determine this:

CMD: node_exporter PID: 3087632 TID: dev OPENFILE: /host/root/var/lib/kubelet/pods/385f109c-132c-4d61-bd6f-bdcc0ba345cf/volumes/kubernetes.io~portworx-volume/pvc-ca28e648-3a66-4bc9-ba8c-3d5d9b313406 START_TIME: Fri Sep  1 04:57:05 2023 RUN_TIME: 05:18 CMDLINE: /bin/node_exporter--path.procfs=/host/proc--path.sysfs=/host/sys--path.rootfs=/host/root--web.listen-address=[0.0.0.0]:8999--log.level=debug--collector.netstat.fields=.*--collector.filesystem.mount-points-exclude=^(.*)--collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

Are these mountpoints btrfs, by any chance?

Sort of. The volumes themselves are XFS/EXT4. But with the storage solution we use, the backend blocks get written to a PURE device through a btrfs filesystem that is mounted on the nodes.

You are possibly hitting #2772. Can you try disabling the btrfs collector temporarily?