Filsystem Device Error on Datasets Mounted Under /home Directory

[norsemangrey]

For some reason I am getting node_export is not able to collect metrics some a couple of my ZFS datasets and they are showing up with error under node_filesystem_device_error metric. Commeon for the once showing up with error is that they are mounted on the /home/user directory. There are no problem with any of the datasets mounted elsewhere.

Host operating system:

Linux Ubuntu 21.04 5.11.0-49-generic x86_64

node_exporter version:

1.2.2

node_exporter command line flags

[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter

[Service]
Type=simple
User=exporters
Group=exporters
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/node_exporter \
    --collector.disable-defaults \
    --collector.cpu \
    --collector.meminfo \
    --collector.netstat \
    --collector.netdev \
    --collector.loadavg \
    --collector.hwmon \
    --collector.diskstats \
    --collector.nvme \
    --collector.zfs \
    --collector.mdadm \
    --collector.filesystem \
    --web.listen-address=:9100 \
    --web.telemetry-path="/metrics"

[Install]
WantedBy=multi-user.target

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Added filesystem collector to command line.

What did you expect to see?

Filesystem metrics for all ZFS datasets.

What did you see instead?

Filesystem metrics for all ZFS datasets except those mounted on /home/user directory.

[discordianfish]

Anything in the logs?

[Lefuneste83]

I have had the same issue today on my Linux Mint workstation while setting up alerting with alertmanager. After a few tries, I came to the conclusion that it simply comes from the fact that the process user of node_exporter (prometheus in my distro), had no read permissions on the file systems and mount points owned by the local user. I have just added the user prometheus to my user group and restarted the prometheus-node-exporter service and all KO are gone. That was pretty obvious now that I think about it...