beszel icon indicating copy to clipboard operation
beszel copied to clipboard
verified publisher icon henrygd

[Bug]: Error getting disk stats

Open techolsy opened this issue 4 months ago • 1 comments

Component

Hub

Description

I am unable to monitor some disks, I get the error Error getting disk stats

Full log here with debug

Aug 24 15:15:42 immich systemd[1]: Started beszel-agent.service - Beszel Agent Service.
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 WARN Data directory not found
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG 0.12.3
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Not monitoring ZFS ARC err="open /proc/spl/kstat/zfs/arcstats: no such file or directory"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Disk partitions="[{\"device\":\"/dev/dm-1\",\"mountpoint\":\"/\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"/dev/dm-3\",\"mountpoint\":\"/opt/immich-app/library\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"/dev/dm-0\",\"mountpoint\":\"/opt/immich-app/psql\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\"]} {\"device\":\"/dev/dm-4\",\"mountpoint\":\"/var/lib/docker\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\"]}]"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Disk I/O diskstats="map[dm-0:{\"readCount\":17456,\"mergedReadCount\":0,\"writeCount\":1255770,\"mergedWriteCount\":0,\"readBytes\":400610304,\"writeBytes\":8219136000,\"readTime\":7952,\"writeTime\":984252,\"iopsInProgress\":0,\"ioTime\":299172,\"weightedIO\":995292,\"name\":\"dm-0\",\"serialNumber\":\"\",\"label\":\"psql-data\"} dm-1:{\"readCount\":47054,\"mergedReadCount\":0,\"writeCount\":2347499,\"mergedWriteCount\":0,\"readBytes\":1641718784,\"writeBytes\":15560491008,\"readTime\":29228,\"writeTime\":1716140,\"iopsInProgress\":0,\"ioTime\":4472904,\"weightedIO\":1752912,\"name\":\"dm-1\",\"serialNumber\":\"\",\"label\":\"vg-root\"} dm-2:{\"readCount\":16314,\"mergedReadCount\":0,\"writeCount\":69754,\"mergedWriteCount\":0,\"readBytes\":70475776,\"writeBytes\":285712384,\"readTime\":5464,\"writeTime\":354800,\"iopsInProgress\":0,\"ioTime\":2208,\"weightedIO\":360264,\"name\":\"dm-2\",\"serialNumber\":\"\",\"label\":\"vg-swap\"} dm-3:{\"readCount\":16346,\"mergedReadCount\":0,\"writeCount\":34616,\"mergedWriteCount\":0,\"readBytes\":438670336,\"writeBytes\":3297202176,\"readTime\":15512,\"writeTime\":103128,\"iopsInProgress\":0,\"ioTime\":29404,\"weightedIO\":120772,\"name\":\"dm-3\",\"serialNumber\":\"\",\"label\":\"apps-library\"} dm-4:{\"readCount\":101279,\"mergedReadCount\":0,\"writeCount\":482507,\"mergedWriteCount\":0,\"readBytes\":2759554048,\"writeBytes\":12347240448,\"readTime\":45280,\"writeTime\":820700,\"iopsInProgress\":0,\"ioTime\":210580,\"weightedIO\":871704,\"name\":\"dm-4\",\"serialNumber\":\"\",\"label\":\"docker-data\"} sda:{\"readCount\":48543,\"mergedReadCount\":15716,\"writeCount\":1816372,\"mergedWriteCount\":739697,\"readBytes\":1793221632,\"writeBytes\":15846203392,\"readTime\":23232,\"writeTime\":1542557,\"iopsInProgress\":0,\"ioTime\":334072,\"weightedIO\":1738168,\"name\":\"sda\",\"serialNumber\":\"0QEMU_QEMU_HARDDISK_drive-scsi0\",\"label\":\"\"} sda1:{\"readCount\":47946,\"mergedReadCount\":15716,\"writeCount\":1816357,\"mergedWriteCount\":739697,\"readBytes\":1773470720,\"writeBytes\":15846203392,\"readTime\":23061,\"writeTime\":1542492,\"iopsInProgress\":0,\"ioTime\":1141740,\"weightedIO\":1573148,\"name\":\"sda1\",\"serialNumber\":\"0QEMU_QEMU_HARDDISK_drive-scsi0\",\"label\":\"\"} sdb:{\"readCount\":74067,\"mergedReadCount\":27523,\"writeCount\":124384,\"mergedWriteCount\":364250,\"readBytes\":2788488192,\"writeBytes\":12347240448,\"readTime\":29017,\"writeTime\":190429,\"iopsInProgress\":0,\"ioTime\":292996,\"weightedIO\":245952,\"name\":\"sdb\",\"serialNumber\":\"0QEMU_QEMU_HARDDISK_drive-scsi1\",\"label\":\"\"} sdc:{\"readCount\":7516,\"mergedReadCount\":9074,\"writeCount\":19301,\"mergedWriteCount\":16923,\"readBytes\":467604480,\"writeBytes\":3297202176,\"readTime\":7162,\"writeTime\":115019,\"iopsInProgress\":0,\"ioTime\":34584,\"weightedIO\":127482,\"name\":\"sdc\",\"serialNumber\":\"0QEMU_QEMU_HARDDISK_drive-scsi2\",\"label\":\"\"} sdd:{\"readCount\":17557,\"mergedReadCount\":143,\"writeCount\":718854,\"mergedWriteCount\":537193,\"readBytes\":429544448,\"writeBytes\":8219136000,\"readTime\":8006,\"writeTime\":710204,\"iopsInProgress\":0,\"ioTime\":304292,\"weightedIO\":803331,\"name\":\"sdd\",\"serialNumber\":\"0QEMU_QEMU_HARDDISK_drive-scsi3\",\"label\":\"\"} sr0:{\"readCount\":71,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":276480,\"writeBytes\":0,\"readTime\":19,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":16,\"weightedIO\":19,\"name\":\"sr0\",\"serialNumber\":\"QEMU_DVD-ROM_QM00001\",\"label\":\"cidata\"}]"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 INFO Detected root device name=dm-1
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 WARN Device not found in diskstats name=apps-library
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 INFO Detected network interface name=eth0 sent=2370640098 recv=4524633718
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG GPU err="no GPU found - install nvidia-smi, rocm-smi, or tegrastats"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Load average 5m=0.01 15m=0.02
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 ERROR Error getting disk stats name=/opt/immich-app/library err="permission denied"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 WARN Invalid disk I/O. Resetting. name=dm-3 read=100804.83 write=757684.97
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 WARN Device not found in diskstats name=apps-library
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Temperature sensors=[]
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG sysinfo data="{Hostname:immich KernelVersion:6.1.0-38-amd64 Cores:4 Threads:4 CpuModel:QEMU Virtual CPU version 2.5+ Uptime:990911 Cpu:25 MemPct:19.1 DiskPct:43.6 Bandwidth:0 AgentVersion:0.12.3 Podman:false GpuPct:0 DashboardTemp:0 Os:0 LoadAvg1:0 LoadAvg5:0.01 LoadAvg15:0.02 BandwidthBytes:0 LoadAvg:[0 0.01 0.02]}"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG System data data="&{Stats:{Cpu:25 MaxCpu:0 Mem:5.87 MemUsed:1.12 MemPct:19.1 MemBuffCache:3.66 MemZfsArc:0 Swap:0.95 SwapUsed:0.07 DiskTotal:8.8 DiskUsed:3.64 DiskPct:43.6 DiskReadPs:0 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[] GPUData:map[] LoadAvg1:0 LoadAvg5:0 LoadAvg15:0 Bandwidth:[0 0] MaxBandwidth:[0 0] LoadAvg:[0 0.01 0.02]} Info:{Hostname:immich KernelVersion:6.1.0-38-amd64 Cores:4 Threads:4 CpuModel:QEMU Virtual CPU version 2.5+ Uptime:990911 Cpu:25 MemPct:19.1 DiskPct:43.6 Bandwidth:0 AgentVersion:0.12.3 Podman:false GpuPct:0 DashboardTemp:0 Os:0 LoadAvg1:0 LoadAvg5:0.01 LoadAvg15:0.02 BandwidthBytes:0 LoadAvg:[0 0.01 0.02]} Containers:[]}"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Containers data="[0xc0000a44d0 0xc0001dafc0 0xc0002da700 0xc0002da7e0]"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Extra FS data=map[apps-library:0xc0001da3f0]
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 DEBUG Stats data="&{Stats:{Cpu:25 MaxCpu:0 Mem:5.87 MemUsed:1.12 MemPct:19.1 MemBuffCache:3.66 MemZfsArc:0 Swap:0.95 SwapUsed:0.07 DiskTotal:8.8 DiskUsed:3.64 DiskPct:43.6 DiskReadPs:0 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[apps-library:0xc0001da3f0] GPUData:map[] LoadAvg1:0 LoadAvg5:0 LoadAvg15:0 Bandwidth:[0 0] MaxBandwidth:[0 0] LoadAvg:[0 0.01 0.02]} Info:{Hostname:immich KernelVersion:6.1.0-38-amd64 Cores:4 Threads:4 CpuModel:QEMU Virtual CPU version 2.5+ Uptime:990911 Cpu:25 MemPct:19.1 DiskPct:43.6 Bandwidth:0 AgentVersion:0.12.3 Podman:false GpuPct:0 DashboardTemp:0 Os:0 LoadAvg1:0 LoadAvg5:0.01 LoadAvg15:0.02 BandwidthBytes:0 LoadAvg:[0 0.01 0.02]} Containers:[0xc0000a44d0 0xc0001dafc0 0xc0002da700 0xc0002da7e0]}"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 WARN Error creating WebSocket client err="HUB_URL environment variable not set"
Aug 24 15:15:42 immich beszel-agent[2499530]: 2025/08/24 15:15:42 INFO Starting SSH server addr=:45876 network=tcp
Aug 24 15:15:48 immich beszel-agent[2499530]: 2025/08/24 15:15:48 INFO SSH connected addr=10.12.14.114:58574
Aug 24 15:15:48 immich beszel-agent[2499530]: 2025/08/24 15:15:48 DEBUG Cached data session=6e6ce7889e3389a8369f6d8f00316be42a13b27d00156b91e4a5212124daa862
Aug 24 15:15:48 immich beszel-agent[2499530]: 2025/08/24 15:15:48 INFO SSH connection established

I also use binary with systemd file and it looks like this

# /lib/systemd/system/beszel-agent.service
[Unit]
Description=Beszel Agent Service
After=network.target

[Service]
Environment="PORT=45876"
Environment="KEY=aaabbbccc"
ExecStart=/usr/local/beszel-agent/beszel-agent
User=beszel
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/beszel-agent.service.d/override.conf
[Service]
Environment="EXTRA_FILESYSTEMS=/opt/immich-app/library,/dev/mapper/apps-library"
Environment="LOG_LEVEL=debug"

It cannot get the path /opt/immich/library because permissions? But it can find see the lv apps-library, but it shows no data

Image

Not really sure what is going on here, or when it stopped working, it have worked before and i have updated beszel many times, but not really caught this issue until today

Extra not, if i monitor another disk but in /etc, like /etc/lib/docker it does work, so i am really confused about this permissions error since the permission on the linux side is the same on both paths

Expected Behavior

I expect to see the volume and how much space its using and how much IO is used

Steps to Reproduce

  1. I showed my systemd service file above, copy and paste should work, cant really reproduce

Category

Metrics

Affected Metrics

Storage

OS / Architecture

Debian GNU/Linux 12 (bookworm)

Beszel version

0.12.3

Installation method

Binary

Configuration

Hub Logs

Agent Logs

techolsy avatar Aug 24 '25 13:08 techolsy

I am having the issue as well running in a docker container, in my case it seems related to the host is running alpine linux which doesn't include OpenZFS, therefore the path /proc/spl/kstat/zfs/arcstats doesn't exist.

omarisai avatar Aug 24 '25 23:08 omarisai

Could it be that the beszel user does not have access to these paths?

svenvg93 avatar Oct 27 '25 19:10 svenvg93

Could it be that the beszel user does not have access to these paths?

It could be that simple, have not really been digging into the problem, strange that it would lose permission over time, but would be something to check out and see if it could be that simple

techolsy avatar Oct 28 '25 08:10 techolsy

@svenvg93 I looked into it more and you was right, i basically only looked at the permission at full path /opt/immich-app/library and the permission was correct there, but also at /opt it was /opt-immich-app that was missing execute permissions for the beszel user, im closing this issue

techolsy avatar Oct 28 '25 17:10 techolsy