btop icon indicating copy to clipboard operation
btop copied to clipboard

[BUG] Disk IO showing only for boot filesystem

Open FlyingHavoc opened this issue 3 years ago • 29 comments

Describe the bug

Disk IO showing only for boot filesystem

Expected behavior

Disk IO should also display IO for / filesystem

See the screenshot on missing IO for /

image

cat /etc/fstab proc /proc proc defaults 0 0 PARTUUID=97306748-01 /boot vfat defaults 0 2 PARTUUID=97306748-02 / ext4 defaults,noatime 0 1

Info (please complete the following information):

  • btop++ version: 1.0.20
  • Binary: static binary
  • Architecture: [x86_64, aarch64, etc.] aarch64
  • Platform: raspberrypi 5.10.76-v8+ aarch64 GNU/Linux
  • (Linux) Kernel: 5.10.76-v8+
  • Os release version: Debian GNU/Linux 10 (buster)

FlyingHavoc avatar Nov 05 '21 14:11 FlyingHavoc

I also get this on a Raspberry Pi 4 4G running 32-bit RPiOS.

Have any idea how to fix this @aristocratos?

Crilum avatar Nov 13 '21 03:11 Crilum

@FlyingHavoc @Crilum For IO stats to work on a device, the dev name needs to be matched with the dev name for the IO stats. Do the following to see if there is IO stats available for your / device.

  1. realpath $(cat /etc/mtab |grep ' / ' | cut -f1 -d" ") to get the canonical path to the device.
  2. ls /sys/block to see if there is folder with the same filename.
  3. Check if the folder contains a stat file and post the contents here.

aristocratos avatar Nov 14 '21 01:11 aristocratos

@aristocratos My output, sort of. As you can see, /sys/block does not contain a folder named root:

pi@raspberrypi:~ $ realpath $(cat /etc/mtab |grep ' / ' | cut -f1 -d" ")
/dev/root
pi@raspberrypi:~ $ ls /sys/block
loop0   loop2  loop5  loop8    ram0   ram11  ram14  ram3  ram6  ram9
loop1   loop3  loop6  loop9    ram1   ram12  ram15  ram4  ram7  sda
loop10  loop4  loop7  mmcblk0  ram10  ram13  ram2   ram5  ram8

That would be the problem, wouldn't it?

Crilum avatar Nov 14 '21 02:11 Crilum

@Crilum try readlink -f $(cat /etc/mtab |grep ' / ' | cut -f1 -d" ") and see if that gives you a real device name

aristocratos avatar Nov 14 '21 11:11 aristocratos

pi@raspberrypi  ~  realpath $(cat /etc/mtab |grep ' / ' | cut -f1 -d" ") /dev/root

pi@raspberrypi  ~  ls /sys/block loop0 loop4 ram0 ram12 ram2 ram6 sda loop1 loop5 ram1 ram13 ram3 ram7 loop2 loop6 ram10 ram14 ram4 ram8 loop3 loop7 ram11 ram15 ram5 ram9

pi@raspberrypi  ~  readlink -f $(cat /etc/mtab |grep ' / ' | cut -f1 -d" ") /dev/root

FlyingHavoc avatar Nov 14 '21 12:11 FlyingHavoc

@aristocratos

So real path is /dev/root But it is /sys/block/sda

There is stat file for it

pi@raspberrypi  ~  ls /sys/block/sda
alignment_offset ext_range ro bdi hidden sda1 capability holders sda2 dev inflight size device mq slaves discard_alignment power stat events queue subsystem events_async range trace events_poll_msecs removable uevent pi@raspberrypi  ~ 

pi@raspberrypi  ~  cat /sys/block/sda/stat 151845 37386 10772583 7833393 7980206 11527309 198463682 28265701 0 8640008 37563584 0 0 0 0 2269705 1464489

FlyingHavoc avatar Nov 14 '21 12:11 FlyingHavoc

@FlyingHavoc Yeah, the issue seems to be that the link /dev/root doesn't resolve down to /dev/sda, so btop has no way of knowing which device to get stats from.

Can possibly add a special case when the device name is /dev/root, but will still need a method to map mountpoint / to it's real device name in that case.

aristocratos avatar Nov 14 '21 12:11 aristocratos

@FlyingHavoc

Yeah, the issue seems to be that the link /dev/root doesn't resolve down to /dev/sda, so btop has no way of knowing which device to get stats from.

Can possibly add a special case when the device name is /dev/root, but will still need a method to map mountpoint / to it's real device name in that case.

@aristocratos Maybe a filter could be used as a workaround for this? Anyway, let me know how I can help. I could test this if you send me debug/test bits. Thanks

FlyingHavoc avatar Nov 14 '21 12:11 FlyingHavoc

@FlyingHavoc This seems specific for raspberry pi's, which I haven't got any to test on. So what would be really helpful would be if you can find some other way than relying on /dev/root being a symlink to find out which device it is using.

What does your output from mount look like?

~~Also, I think a workaround for now would be to change this line in your fstab: PARTUUID=97306748-02 / ext4 defaults,noatime 0 1 to: /dev/sda2 / ext4 defaults,noatime 0 1 (if /dev/sda2 is your root partition?) and make sure use fstab is true in the btop options. Should work.~~ Edit: Realized it only gets the mountpoint from fstab and not the device name, only /etc/mtab or /proc/self/mounts are used for device name.

aristocratos avatar Nov 14 '21 13:11 aristocratos

@aristocratos

pi@raspberrypi  ~  mount
/dev/sda2 on / type ext4 (rw,noatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=3834052k,nr_inodes=958513,mode=755) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,relatime)

FlyingHavoc avatar Nov 14 '21 13:11 FlyingHavoc

@aristocratos

root@raspberrypi:/dev# cd /sys/block/ root@raspberrypi:/sys/block# l total 0 .. lrwxrwxrwx 1 root root 0 Nov 14 20:27 sda -> ../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/block/sda

root@raspberrypi:/sys/block# l ../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/block/sda total 0 -r--r--r-- 1 root root 4096 Nov 14 21:03 alignment_offset lrwxrwxrwx 1 root root 0 Nov 14 21:03 bdi -> ../../../../../../../../../../../../../../virtual/bdi/8:0 -r--r--r-- 1 root root 4096 Nov 14 21:03 capability -r--r--r-- 1 root root 4096 Nov 14 20:27 dev lrwxrwxrwx 1 root root 0 Nov 14 21:03 device -> ../../../0:0:0:0 -r--r--r-- 1 root root 4096 Nov 14 21:03 discard_alignment -r--r--r-- 1 root root 4096 Nov 14 21:03 events -r--r--r-- 1 root root 4096 Nov 14 21:03 events_async -rw-r--r-- 1 root root 4096 Nov 14 21:03 events_poll_msecs -r--r--r-- 1 root root 4096 Nov 14 21:03 ext_range -r--r--r-- 1 root root 4096 Nov 14 21:03 hidden drwxr-xr-x 2 root root 0 Nov 14 21:03 holders -r--r--r-- 1 root root 4096 Nov 14 21:03 inflight drwxr-xr-x 3 root root 0 Nov 14 21:03 mq drwxr-xr-x 2 root root 0 Nov 14 21:03 power drwxr-xr-x 3 root root 0 Nov 14 20:27 queue -r--r--r-- 1 root root 4096 Nov 14 21:03 range -r--r--r-- 1 root root 4096 Nov 14 21:03 removable -r--r--r-- 1 root root 4096 Nov 14 21:03 ro drwxr-xr-x 5 root root 0 Nov 14 20:25 sda1 drwxr-xr-x 5 root root 0 Nov 14 20:25 sda2 -r--r--r-- 1 root root 4096 Nov 14 20:27 size drwxr-xr-x 2 root root 0 Nov 14 21:03 slaves -r--r--r-- 1 root root 4096 Nov 14 20:53 stat lrwxrwxrwx 1 root root 0 Nov 14 21:03 subsystem -> ../../../../../../../../../../../../../../../class/block drwxr-xr-x 2 root root 0 Nov 14 21:03 trace -rw-r--r-- 1 root root 4096 Nov 14 21:03 uevent

root@raspberrypi:/sys/block# cat ../devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/block/sda/stat 152042 37386 10790600 7833626 8164769 11782450 202686666 28375019 0 8814312 37704351 0 0 0 0 2320866 1495705

FlyingHavoc avatar Nov 14 '21 18:11 FlyingHavoc

@FlyingHavoc Are you using a USB or Hard Drive to boot? In case you didn't know (no offence intended), sda stands for "Special Device A", sdb for "Special Device B", and so on and so forth. I don't think the system labels the boot drive as a Special Device, so that might not be the right drive. :smile: My root drive is an SD card, and it looks something like this:

pi@raspberrypi:~ $ cd /sys/block/mmcblk0 && ls
alignment_offset  discard_alignment  force_ro   mmcblk0p2  removable  subsystem
bdi               events             hidden     mq         ro         trace
capability        events_async       holders    power      size       uevent
dev               events_poll_msecs  inflight   queue      slaves
device            ext_range          mmcblk0p1  range      stat
pi@raspberrypi:/sys/block/mmcblk0 $ cat /sys/block/mmcblk0/stat
   28466    16509  2267929    97579    45943    71435  7223530  3308383        0   378490  3408058      470      135 80959235     2095        0        0
pi@raspberrypi:/sys/block/mmcblk0 $

Also, /dev/root does not exist on my system, despite / being symlinked to /dev/root..

Crilum avatar Nov 15 '21 15:11 Crilum

@Crilum Yes, I'm using USB SSD to boot my pi4. Why one wouldn't? (It gives incredible speed & IO performance, at the same time it doesn't degrade and fail like SD cards do) I'm aware of sdX type of devices ;) (none taken)

The same for my pi4 -- /dev/root does not exist

But somehow it is listed in # cat /etc/mtab /dev/root / ext4 rw,noatime 0 0 and # cat /proc/self/mounts /dev/root / ext4 rw,noatime 0 0

@aristocratos is there a way to map /dev/root or / to /sys/block/sda cat /sys/block/sda/stat 151845 37386 10772583 7833393 7980206 11527309 198463682 28265701 0 8640008 37563584 0 0 0 0 2269705 1464489

FlyingHavoc avatar Nov 26 '21 16:11 FlyingHavoc

@FlyingHavoc

is there a way to map /dev/root or / to /sys/block/sda

Well to summarize, usually /dev/root is a symlink to the actual block device (and would then work as expected since btop finds the origin of the link) but for some reason this isn't the case on pi 4. I don't know if this differs on different distros or if it's specific to the hardware.

But as I mentioned earlier, I don't have any raspberry pi hardware, so finding a connection between /dev/root and it's actual block device besides using the symlink method is not something I can do easily since I wouldn't know if it would look the same on a pi.

aristocratos avatar Nov 26 '21 19:11 aristocratos

With your explanation, I just made a sym-link from /sys/block/mmcblk0 (my block device) to /dev/root/ and now it works.

Fixed with this:

sudo ln -s /sys/block/mmcblk0 /dev/root

btop-shows-io

Crilum avatar Nov 26 '21 19:11 Crilum

@aristocratos @Crilum I've made a symlink too and it's working!

$ sudo ln -s /sys/block/sda /dev/root $ ls -al /dev/root lrwxrwxrwx 1 root root 14 Nov 26 22:27 /dev/root -> /sys/block/sda

image

FlyingHavoc avatar Nov 26 '21 19:11 FlyingHavoc

@FlyingHavoc @Crilum I'm happy it's working for you, but I wouldn't recommend manually linking /dev/root without knowing why the system isn't doing this itself, could be there's a reason for it.

I also believe this link will be reset on a reboot, so would need to script it to do so at every boot for it to continue to work.

aristocratos avatar Nov 26 '21 19:11 aristocratos

If you know of any pi system developers feel free to tag them in the thread, would like to find a fix that doesn't involve the user doing possibly dangerous fixes, for example if linking to the wrong block device :\

aristocratos avatar Nov 26 '21 19:11 aristocratos

Well, my problem of not displaying IO is different from the above。

# cat /etc/mtab 
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
...
/dev/sda3 / xfs rw,relatime,attr2,inode64,noquota 0 0

# realpath $(cat /etc/mtab |grep ' / ' | cut -f1 -d" ")
/root/rootfs
/dev/sda3

On Centos7 system, there will be rootfs information in mtab, so the / directory reading shows two, how to solve this?

xhsky avatar Dec 20 '21 02:12 xhsky

@aristocratos moving the discussion to the correct issue.

To catch up everyone else:

The solution, at least for ARM devices, seems to be parsing /proc/cmdline. From memory the syntax is rootfs=/dev/path,mount,opts.

I'm not sure when exactly, but I'll grab a dump from my device (not a Pi, but still an ARM SoC) and comment here.

jaskij avatar May 11 '22 19:05 jaskij

Checked it, and I was wrong (that's what I get for writing from memory). There are no mounting options in root=.

My i.MX8X device has this as a commandline

console=ttyLP2,115200 earlycon root=/dev/mmcblk1p2 rootwait rw

For comparison, my PC system (Manjaro converted to Arch) has:

initrd=\amd-ucode.img initrd=\initramfs-linux-lts.img root="LABEL=arch_root" rw

jaskij avatar May 12 '22 18:05 jaskij

/proc/cmdline from a Raspberry Pi 4 running Raspberry Pi OS 11 (Bullseye):

$ cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1  smsc95xx.macaddr=DC:A6:32:68:B7:6D vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS0,115200 console=tty1 root=PARTUUID=9c72054b-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles

Crilum avatar May 12 '22 19:05 Crilum

uh, sorry to ping everyone in this thread, but did the solution of parsing the /proc/cmdline actually got implemented? I am using the latest 1.2.13 on raspberry pi 4. No IO graph except for /boot is abailable, which is not quite useful. Heres my cmdline:

coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=HDMI-A-1:3840x2160M@30 smsc95xx.macaddr=E4:5F:01:03:FD:A8 vc_mem.mem_base=0x3eb00000 vc_mem.mem_size=0x3ff00000  console=ttyS0,115200 console=tty1 root=PARTUUID=b926a161-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles

notice that the root= part is not actuallt referencing a path. maybe that is an issue? if so should I open a new issue?

SodaWithoutSparkles avatar Nov 16 '22 17:11 SodaWithoutSparkles

@SodaWithoutSparkles I don't think there was ever a fix implemented..

Like I said above, you can temporarily fix this by running sudo ln -s /sys/block/mmcblk0 /dev/root (very hacky fix, may have side effects..), replacing the root block device with your own, but it will reset after a reboot.

Crilum avatar Nov 16 '22 19:11 Crilum

I don't think the cmdline not having an actual path effects the root io.

Here's the output of cat /proc/cmdline from my x86_64 PC, and it doesn't have a path either, but the root io in btop works:

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.10.0-19-amd64 root=UUID=68adc7c1-e216-40b6-b9d2-d91894b9af16 ro quiet

Crilum avatar Nov 16 '22 19:11 Crilum