node_filesystem collector unable deduplicate data from multihomed mounts like nfs

[zeronewb]

Host operating system: output of uname -a

3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.4.0 (branch: HEAD, revision: 7da1321761b3b8dfc9e496e1a60e6a476fec6018)

node_exporter command line flags

/usr/sbin/node_exporter --collector.textfile.directory /var/lib/node_exporter/textfile_collector --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

node_exporter log output

Oct 21 19:34:07 node_exporter: level=error ts=2022-10-21T23:34:07.079Z caller=stdlib.go:105 msg="error gathering metrics: 7 error(s) occurred:\n* [from Gatherer #2] collected metric "node_filesystem_device_error" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_size_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.36870912e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_free_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.2514783232e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_avail_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.2514783232e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_files" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:1.048576e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_files_free" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:1.048568733e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_readonly" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:0 > } was collected before with the same name and label values"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

SAN has multiple A records for load balancing. /proc/self/mounts storagesystem:/exports/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.1.1.1,mountvers=3,mountport=300,mountproto=tcp,local_lock=all,addr=1.1.1.1 0 0 storagesystem:/exports/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.1.1.2,mountvers=3,mountport=300,mountproto=tcp,local_lock=all,addr=1.1.1.2 0 0

What did you expect to see?

Deduplicate data from /proc based on local mounts, and iterate correctly.

What did you see instead?

Log errors above.

[discordianfish]

Ugh yeah that's annoying. Also feels like this might be a regression. I think we had an issue (and fix) for this before? @SuperQ?

[rexagod]

Not sure if there's a fix for this, but I can take this up if there isn't. :)