disk read/write rates reported much too high

[thomas725]

SUMMARY

packaged dstat reports correctly between 106-122mb/s for my usb3 disk under load. dool installed via AUR package dool-git reports 857-1030mb/s which is clearly not the rate the actual copy process is going.

ISSUE TYPE

• Bug Report

DSTAT VERSION

$ dstat --version
Dstat 0.8.0
Written by Dag Wieers <[email protected]>
Homepage at http://dag.wieers.com/home-made/dstat/

Platform posix/linux
Kernel 5.10.53-1-MANJARO
Python 3.9.6 (default, Jun 30 2021, 10:22:16) 
[GCC 11.1.0]

Terminal type: xterm-256color (color support)
Terminal size: 34 lines, 236 columns

Processors: 12
Pagesize: 4096
Clock ticks per secs: 100

internal:
        aio,cpu,cpu-adv,cpu-use,cpu24,disk,disk24,disk24-old,epoch,fs,int,int24,io,ipc,load,lock,mem,mem-adv,net,page,page24,proc,raw,socket,swap,swap-old,sys,tcp,time,udp,unix,vm,vm-adv,zones
/usr/share/dstat:
        battery,battery-remain,condor-queue,cpufreq,dbus,disk-avgqu,disk-avgrq,disk-svctm,disk-tps,disk-util,disk-wait,dstat,dstat-cpu,dstat-ctxt,dstat-mem,fan,freespace,fuse,gpfs,gpfs-ops,helloworld,ib,
        innodb-buffer,innodb-io,innodb-ops,jvm-full,jvm-vm,lustre,md-status,memcache-hits,mongodb-conn,mongodb-mem,mongodb-opcount,mongodb-queue,mongodb-stats,mysql-io,mysql-keys,mysql5-cmds,mysql5-conn,mysql5-innodb,
        mysql5-innodb-basic,mysql5-innodb-extra,mysql5-io,mysql5-keys,net-packets,nfs3,nfs3-ops,nfsd3,nfsd3-ops,nfsd4-ops,nfsstat4,ntp,postfix,power,proc-count,qmail,redis,rpc,rpcd,sendmail,snmp-cpu,snmp-load,
        snmp-mem,snmp-net,snmp-net-err,snmp-sys,snooze,squid,test,thermal,top-bio,top-bio-adv,top-childwait,top-cpu,top-cpu-adv,top-cputime,top-cputime-avg,top-int,top-io,top-io-adv,top-latency,top-latency-avg,
        top-mem,top-oom,utmp,vm-cpu,vm-mem,vm-mem-adv,vmk-hba,vmk-int,vmk-nic,vz-cpu,vz-io,vz-ubc,wifi,zfs-arc,zfs-l2arc,zfs-zil

DOOL VERSION

$ dool --version
Dool 0.9.10
Written by Scott Baker <[email protected]>
Forked from Dstat written by Dag Wieers <[email protected]>
Homepage at https://github.com/scottchiefbaker/dool/

Platform posix/linux
Kernel 5.10.53-1-MANJARO
Python 3.9.6 (default, Jun 30 2021, 10:22:16) 
[GCC 11.1.0]

Terminal type: xterm-256color (color support)
Terminal size: 34 lines, 236 columns

Processors: 12
Pagesize: 4096
Clock ticks per secs: 100

internal:
        aio,cpu,cpu-adv,cpu-use,cpu24,disk,disk24,disk24-old,epoch,fs,int,int24,io,ipc,load,lock,mem,mem-adv,net,page,page24,proc,raw,socket,swap,swap-old,sys,tcp,time,udp,unix,vm,vm-adv,zones
/usr/share/dool:
        battery,battery-remain,condor-queue,cpufreq,dbus,disk-avgqu,disk-avgrq,disk-svctm,disk-tps,disk-util,disk-wait,dool,dool-cpu,dool-ctxt,dool-mem,fan,freespace,fuse,gpfs,gpfs-ops,helloworld,ib,
        innodb-buffer,innodb-io,innodb-ops,jvm-full,jvm-vm,lustre,md-status,memcache-hits,mongodb-conn,mongodb-mem,mongodb-opcount,mongodb-queue,mongodb-stats,mysql-io,mysql-keys,mysql5-cmds,mysql5-conn,mysql5-innodb,
        mysql5-innodb-basic,mysql5-innodb-extra,mysql5-io,mysql5-keys,net-packets,nfs3,nfs3-ops,nfsd3,nfsd3-ops,nfsd4-ops,nfsstat4,ntp,postfix,power,proc-count,qmail,redis,rpc,rpcd,sendmail,snmp-cpu,snmp-load,
        snmp-mem,snmp-net,snmp-net-err,snmp-sys,snooze,squid,test,thermal,top-bio,top-bio-adv,top-childwait,top-cpu,top-cpu-adv,top-cputime,top-cputime-avg,top-int,top-io,top-io-adv,top-latency,top-latency-avg,
        top-mem,top-oom,utmp,vm-cpu,vm-mem,vm-mem-adv,vmk-hba,vmk-int,vmk-nic,vz-cpu,vz-io,vz-ubc,wifi,zfs-arc,zfs-l2arc,zfs-zil

OS / ENVIRONMENT

Manjaro (Arch based)

STEPS TO REPRODUCE

run dstat and dool (with parameters that make it show disk transfer rates) and compare their outputs, for example: dool -tafys and dstat -tafmspl

EXPECTED RESULTS

matching realistic values for disk read/write rates

ACTUAL RESULTS

$ dool -taf --debug
Plugins: ['time', 'cpu', 'disk', 'net', 'page', 'mem', 'sys', 'proc', 'load']
Module dool_time
Module dool_cpu requires ['/proc/stat']
Module dool_disk requires ['/proc/diskstats']
Module dool_net requires ['/proc/net/dev']
Module dool_page requires ['/proc/vmstat']
Module dool_mem requires ['/proc/meminfo']
Module dool_sys requires ['/proc/stat']
Module dool_proc requires ['/proc/stat']
Module dool_load requires ['/proc/loadavg']
-------system------ --total-cpu-usage-- dsk/nvme0n1---dsk/sda-----dsk/sdb-- --net/tun0--net/wlp59s0 ---paging-- ------memory-usage----- ---system-- ---procs--- ---load-avg---
        time       |usr sys idl wai stl| read  writ: read  writ: read  writ| recv  send: recv  send|  in   out | used  free  buff  cach| int   csw |run blk new| 1m   5m  15m 
Aug-25 02:52:23.421| 46  18  36   1   0| 900k 8288k: 424k 2414k:8606k 1203b|   0     0 :   0     0 |   0     6B|23.1G  498M 66.9M 44.9G|3649    11k|3.0   0 7.1|3.51 3.22 4.15  0.94ms
Aug-25 02:52:24.422|  9   4  85   2   0|   0     0 :   0     0 : 971M    0 |   0     0 :  45k   15k|   0     0 |23.1G  487M 66.9M 44.9G|9708    18k|2.0 1.0  13|3.51 3.22 4.15  0.94ms
Aug-25 02:52:25.422| 10   4  84   2   0|   0     0 :   0     0 : 997M    0 |   0     0 :  60k   28k|   0     0 |23.1G  482M 66.9M 44.9G|  12k   18k|2.0   0  15|3.51 3.22 4.15  1.35ms
Aug-25 02:52:26.422|  8   5  85   2   0|   0     0 :   0     0 :1030M    0 |   0     0 :  59k   28k|   0     0 |23.1G  471M 66.9M 44.9G|  12k   19k|2.0   0  20|3.51 3.22 4.15  3.21ms

[pezacz]

I have the same issue on QNAP. It looks like mbit/s is printed instead MB/s

[SuperTux88]

I had the same problem (and it's not only disks, it's also network). It's because the defaults were changed (see e860706dda71498f1d8fa4fa10f3f05322837f45 and also first as driveby here: 573843108557f260d0e657c6cf4fec8bf66b5ef8), dstat used bytes by default (and has a --bits option to change that), dool now uses bits by default and has an undocumented --bytes option to switch back to bytes. The help still only lists the --bits option to force to bits.

I don't know why this was changed (I didn't find more background information about that change), I personally found the bytes-default much more useful.

[rklasen]

I can confirm the issue, and it seems the network transfer speed is also affected. The option --bytes indeed solves this, but I would love for this option to be the default as well, since that was the case of dstat.

[jkleckner]

@scottchiefbaker Could you elaborate on the reason for switching the default from bytes to bits?

[scottchiefbaker]

All the network speeds I deal with on a regular basis are reported in bits. It only made sense for my fork to make --bits the default. Since I primarily use Dool to report on network usage, and it's impact on disk performance it made sense.

The --bytes option is available if you need output in XB/s.

[pezacz]

bits/s are very practical at these times when my home NAS does 8589934592 bits per second in peak. It feels more satisfying, more powerful then small numbers using such insignificant units like Bytes. And especially its delightful to practice calculations when you interested to read-out size of your IO operations. Its easy. Just divide by 2 and 2 and ... 2 again and all is just fine. Only small kids can't do that on the fly watching columns of numbers rolling...

I know only one better unit to use. Sectors of preferably unknown size and let anyone look it-up how many bytes/sector do we talk about. Like memory pages in OS.

End of sarcasm - in 2022 hardly anyone need to measure bits/second, because we are dealing with hundreds of millions of Bytes/second way, way more often, then bits/second. Networking is using bits, because serial communication 40Y ago and because its cooler when IS offers 40mbit connection then 5MB connection. Bigger numbers sell better, right. Please, consider that. If you in doubt what people prefer, there is always https://doodle.com And if you don't care well - then you don't ;)

[jkleckner]

@scottchiefbaker Thanks for updating the doc on --bytes.

I'm grateful for your contributions here.

[SuperTux88]

All the network speeds I deal with on a regular basis are reported in bits.

While network speeds are often advertised in bits, that's not the case for disk speeds, and having the disk speeds reported in bits by default is very confusing and unexpected (and the fact that an issue like this exists proves that there are multiple people confused by this). Also, most UIs also report network speeds (download speeds for example) in bytes, so most people should be used to see speeds in bytes. That's why I think bytes would still be the better default. I think way more people would need to use a --bytes parameter to switch back to the old default, than people had to use a --bits parameter before.

Also, the manpage still only talks about the --bits parameter and doesn't mention --bytes.

[jkleckner]

Personally, I also like network speeds in bits but disk I/O in bytes. That would be my ideal.

[malventano]

Another vote for bytes being the default, as it is in line with the long-standing expected behavior and confuses those moving from dstat to dool. No other disk monitoring tool defaults to bits per second.

[scottchiefbaker]

For now I'm going to leave bits as the default. The --bytes option is documented and an easy alternative.