node_exporter endpoint timeout, couldn't get dbus connection: write unix @->/run/systemd/private: write: broken pipe

[samtangweicheng]

node exporter endpoint timeout on my 2 machines, and their log shows systemd collector fail to write unix file. After restart, noded exporter is able to resume normal operation, but endpoint timeout would reappears a few hours later.

Host operating system: output of uname -a

Linux ydcs-vmziksyzjb 5.4.224-1.el7.elrepo.x86_64 #1 SMP Tue Nov 8 17:24:56 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.4.1 (branch: HEAD, revision: a954c9fea6bcccea7f68a77a002a2a878ef1d60c) build user: root@0cb1bc5fbc4c build date: 20221129-18:45:59 go version: go1.19.3 platform: linux/amd64

node_exporter command line flags

/usr/local/node_exporter/node_exporter/node_exporter --web.listen-address=:19101 --collector.systemd --web.config=/usr/local/node_exporter/node_exporter/node_exporter.yml --collector.textfile.directory=/usr/local/node_exporter/node_exporter/textfile.collected

other information: systemd version

systemd 243 (v243-31.ctl2) +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy

last reboot: reboot system boot 5.4.224-1.el7.el Mon May 15 19:17 still running reboot system boot 5.4.224-1.el7.el Thu May 11 09:42 still running reboot system boot 5.4.224-1.el7.el Thu Mar 30 15:54 still running reboot system boot 5.4.224-1.el7.el Thu Mar 30 11:48 - 15:54 (04:06) reboot system boot 5.4.224-1.el7.el Mon Mar 6 10:14 - 11:48 (24+01:33)

node_exporter log output

Nov 19 12:48:02 ydcs-vmziksyzjb systemd[1]: Started node_exporter. Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.989Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.4.1, branch=HEAD, revision=a954c9fea6bcccea7f68a77a002a2a878ef1d60c)" Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.989Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.19.3, user=root@0cb1bc5fbc4c, date=20221129-18:45:59)" Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.989Z caller=node_exporter.go:185 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required." Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=systemd_linux.go:152 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=.+ Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=systemd_linux.go:154 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+.(automount|device|mount|scope|slice) Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|v> Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|> Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:108 level=info msg="Enabled collectors" Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=arp Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=bcache Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=bonding Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=btrfs Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=conntrack Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=cpu Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=cpufreq Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=diskstats Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=dmi Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=edac Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=entropy Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=fibrechannel Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=filefd Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=filesystem Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=hwmon Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=infiniband Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=ipvs Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=loadavg Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=mdadm Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=meminfo Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=netclass Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=netdev Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=netstat Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=nfs Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=nfsd Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=nvme Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=os Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=powersupplyclass Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=pressure Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=rapl Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=schedstat Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=selinux Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=sockstat Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=softnet Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=stat Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=systemd Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=tapestats Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=textfile Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=thermal_zone Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=time Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=timex Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=udp_queues Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=uname Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=vmstat Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=xfs Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=zfs Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:199 level=info msg="Listening on" address=:19101 Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=tls_config.go:231 level=info msg="TLS is disabled." http2=false

Nov 20 00:05:43 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T16:05:43.723Z caller=collector.go:169 level=error msg="collector failed" name=systemd duration_seconds=240.056944888 err="couldn't get dbus connection: write unix @->/run/systemd/private: write: broken pipe" Nov 20 00:07:43 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T16:07:43.769Z caller=collector.go:169 level=error msg="collector failed" name=systemd duration_seconds=240.071481612 err="couldn't get dbus connection: write unix @->/run/systemd/private: write: broken pipe"

Are you running node_exporter in Docker?

no

What did you do that produced an error?

restart node_exporter, running serval hours, this problem occur every time.

What did you expect to see?

What did you see instead?