raspberrypi
Degraded network performance after recent update
Describe the bug I'm using a Pi 4 as a basic Samba share for several months now. Recently after performing a system update & upgrade I noticed a degraded read and write performance.
It must be related to a recent update since otherwise, nothing has changed. The write performance has dropped from a constant 113MB/s to around 75MB/s.
Since nothing else has changed in my setup I flashed an older build of Raspberry Pi OS ( 64bit was the only image I could find) to the Pi. With this version, it's performing very well, just like the last few months.
But after running a system update && upgrade performance is significantly worse again. The Samba version is exactly the same in both situations.
I guess it's related to the newer kernel and maybe some changes to the underlying network stack?
To reproduce Configure a basic samba share on some external storage with an older version of Raspberry Pi OS. I've used this build
After running a system update && upgrade the write performance is much lower than before.
Expected behaviour
More or less stable transfer at 100MB/s+
Actual behaviour
After a few seconds, the write speed to the Pi drops to around 75MB/s and stays there.
System
Raspberry Pi 4 2GB with Crucial MX500 attached via UASP USB3 adapter. The disk is formatted as XFS and has trim support enabled.
WORKING
Linux raspberrypi 5.4.51-v8+ #1333 SMP PREEMPT Mon Aug 10 16:58:35 BST 2020 aarch64 GNU/Linux
Raspberry Pi reference 2020-08-20
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7252c154838ec5b4576f29c996ac8fe3750cae12, stage4
Aug 19 2020 17:38:16
Copyright (c) 2012 Broadcom
version e90cba19a98a0d1f2ef086b9cafcbca00778f094 (clean) (release) (start)
NOT WORKING
Linux raspberrypi 5.10.11-v7l+ #1399 SMP Thu Jan 28 12:09:48 GMT 2021 armv7l GNU/Linux
Raspberry Pi reference 2021-01-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 21090519d85bdaa1615d5d5057d37b09368ea5d2, stage2
Jan 27 2021 22:19:57
Copyright (c) 2012 Broadcom
version 99d9a48302e4553cff3688692bb7e9ac760a03fa (clean) (release) (start)
Logs No relevant output in dmesg or syslog
Additional context I've tested it with three different drives, and two different Pis. I also reinstalled Raspberry Pi OS several times, with nothing other than samba installed. Therefore I'm 100% sure it's related to one of the updates.
The disk itself is performing well in both cases.
root@raspberrypi/home # dd if=/dev/zero of=test.data bs=4M count=1K
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0589 s, 329 MB/s
Could you please update your issue with details for the used UASP USB3 adapter from dmesg? Background: certain SATA2USB chipset & firmware versions seem to have bugs that impact performance negatively.
example for detail I'm looking for (note: this device is not affected!) [ 3.013976] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd [ 3.045420] usb 2-2: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00 [ 3.053773] usb 2-2: New USB device strings: Mfr=2, Product=3, SerialNumber=1 [ 3.061033] usb 2-2: Product: IB-RD3662-C31 [ 3.065324] usb 2-2: Manufacturer: ICY BOX [ 3.069496] usb 2-2: SerialNumber: 123456789100
and output of lsusb
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
I've attached the output below. At least one guy is experiencing the same issue.
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[ 1.793617] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[ 1.976374] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21
[ 1.976424] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 1.976458] usb 1-1: Product: USB2.0 Hub
[ 1.979228] hub 1-1:1.0: USB hub found
[ 1.979571] hub 1-1:1.0: 4 ports detected
[ 2.124045] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[ 2.137014] systemd[1]: System time before build time, advancing clock.
[ 2.155349] usb 2-1: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00
[ 2.155393] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[ 2.155427] usb 2-1: Product: Ugreen Storage Device
[ 2.155458] usb 2-1: Manufacturer: Ugreen
[ 2.155487] usb 2-1: SerialNumber: 091912070139
[ 2.182292] scsi host0: uas
[ 2.184443] scsi 0:0:0:0: Direct-Access CT250MX5 00SSD1 0 PQ: 0 ANSI: 6
[ 2.187262] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/233 GiB)
[ 2.187500] sd 0:0:0:0: [sda] Write Protect is off
[ 2.187537] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 2.187922] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.188686] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[ 2.235616] sda: sda1
[ 2.238716] sd 0:0:0:0: [sda] Attached SCSI disk
Update 10Feb2021 16:49 UTC I've collected performance metrics using CrystalDiskMark previously and can NOT confirm a significant drop after upgrade to kernel 5.10.11 for my openmediavault based NAS (i.e. read of 64MB file increased from 93 MB/s to 109 MB/s) on the same hardware & setup (details (incl SAMBA config) here)
64MB file? You're not testing the throughput to your disk, you're testing your cache.
Try again with a larger file and several read/ write operations.
I just tested it again.
Flashed the newest Raspberry Pi OS 32bit version (still running the old 5.4.83-v7l+ Kernel ), configured a basic samba share and ran a few tests with around 10-20GB of data. No issues at all, the transfer speed was at 110MB/s all the time.
After that, I updated to the kernel version 5.10.11-v7l+ and updated the preinstalled packages as well. I then ran exactly the same test again, nothing else changed. And the transfer speed dropped after around 3-4GB from 110MB/s to 75MB/s.
This confirms that it's not related to a config issue or some other factor. I ran exactly the same test again, the only thing that has changed is the kernel version.
Can you test if the issue is samba related or network related? E.g. measure network speed with iperf between the two machines?
It doesn't seem to be network related. iPerf shows a constant 950mbit/s in both directions.
Edit: Samba itself shouldn't be the issue either, since the version hasn't changed with the update.
I am having a similar issue with my NFS shares that I am using for my kodi box. Everything was working just fine, but at the moment the video playback is impossible. My rpi4 is currently running: Linux 5.10.14-v7l+ #1401 SMP Mon Feb 8 14:27:07 GMT 2021 armv7l
Before the update everything was working fine. I even turned on my old rpi3 board with :5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l and just copied over my /etc/exports. Suddenly kodi went back to life.
Strangely enough, when accessing the share from my windows laptop, on both the video playback is smooth as butter.
This is strange as for example iperf3 shows stable 900 Mb/s+.
For a test I have played the same movie from both machines, while it played just fine from the RPI3 board, the RPI4 is just full of stutter and after a while it crushes.
RPI4 nfsstat after 2min:
Server rpc stats:
calls badcalls badfmt badauth badclnt
61554 0 0 0 0
Server nfs v3:
null getattr setattr lookup access
7 0% 436 0% 0 0% 28117 45% 194 0%
readlink read write create mkdir
14 0% 32737 53% 0 0% 0 0% 0 0%
symlink mknod remove rmdir rename
0 0% 0 0% 0 0% 0 0% 0 0%
link readdir readdirplus fsstat fsinfo
0 0% 0 0% 39 0% 1 0% 8 0%
pathconf commit
RPI3 after 2 min playback:
Server rpc stats:
calls badcalls badfmt badauth badclnt
1698 0 0 0 0
Server nfs v3:
null getattr setattr lookup access
1 0% 5 0% 0 0% 97 5% 3 0%
readlink read write create mkdir
0 0% 1589 93% 0 0% 0 0% 0 0%
symlink mknod remove rmdir rename
0 0% 0 0% 0 0% 0 0% 0 0%
link readdir readdirplus fsstat fsinfo
0 0% 0 0% 2 0% 0 0% 1 0%
pathconf commit
0 0% 0 0%
I have the same problem with my Pi 4 4GB!
I use it as a NAS with an 8TB WD Elements attached. Performance is much lower after the recent updates
This is also the second time I'm noticing IO errors after the kernel update. I've never seen this issue in the past 6 months with the old kernel.
[59761.546841] sd 1:0:0:0: [sdb] Attached SCSI disk
[152744.813849] XFS (sda1): log I/O error -5
[152744.813945] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1196 of file fs/xfs/xfs_log.c. Return address = 78401cd3
[152744.813963] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[152744.813996] XFS (sda1): Please unmount the filesystem and rectify the problem(s)
@popcornmix Thanks, it looks like this is just the case here. Meanwhile I reverted kernel to the same that was working on rpi3 and magic happened, everything works now.
Same bug.
Upgrated to kernel 5.10.11 speed: 70 MB/s. Returning to 5.4.79-v71+ #1373, speed: 115 MB/s
Pi 4B 2GB + WD Elements 5TB USB 3.0
Upgrated to kernel 5.10.11 speed: 70 MB/s.
New test: It is not the kernel, it is another update.
Continuous speed problem, after updating everything except the Kernel.
My OMV, is working with an old kernel, and after updating everything else, the problem starts.
Oh.. I was quite sure it’s related to the kernel. I’ll have a look at the packages that will get updated
Confirming exactly which package update causes the reduced performance would be helpful.
I'll try to see which package reduces the performance. After a clean install, the following packages have an upgrade available, so one of them must be the problem.
base-files bluez-firmware ca-certificates device-tree-compiler file firmware-atheros firmware-brcm80211
firmware-libertas firmware-misc-nonfree firmware-realtek iproute2 libgnutls30 libldap-2.4-2 libldap-common
libmagic-mgc libmagic1 libpam-systemd libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0
libsystemd0 libudev1 libzstd1 python-rpi.gpio raspberrypi-bootloader raspberrypi-kernel raspberrypi-sys-mods
raspi-config rpi-eeprom rpi.gpio-common sudo systemd systemd-sysv tzdata udev unzip
Edit: Looks like it's indeed not related to the kernel. I've only upgraded raspberrypi-kernel and so far it seems to be stable.
@popcornmix Do you have an idea which of those packages might have an impact? So I don't have to test 35 packages individually.
This combo has the bug.
In a Fresh Install, if you update this:
- bootloader
- kernel
- sys-mods
- raspi-config
- rpi-eeprom
When the server is restarted and changes are applied, then the speed is reduced.
We have to do the test by omitting the SYS-Mod, which is the only one with a description that mentions the performance of the system.
I don't think it's one of those packages. I've just updated them one by one and rebootet but the performance is fine.
So it leaves those packages:
base-files bluez-firmware ca-certificates device-tree-compiler file firmware-atheros firmware-brcm80211
firmware-libertas firmware-misc-nonfree firmware-realtek iproute2 libldap-2.4-2 libldap-common libmagic-mgc
libmagic1 libpam-systemd libsystemd0 libudev1 libzstd1 python-rpi.gpio rpi.gpio-common sudo systemd systemd-sysv
tzdata udev unzip
I think you should be able to rule out the following by upgrading them all together:
bluez-firmware device-tree-compiler firmware-atheros firmware-brcm80211
firmware-libertas firmware-misc-nonfree firmware-realtek python-rpi.gpio rpi.gpio-common tzdata unzip
Test 3. Test performed in a completely new environment.
The following additional packages will be installed:
- libraspberrypi-bin
- libraspberrypi-dev
- libraspberrypi-doc
- libraspberrypi0
The following packages will be upgraded:
- libraspberrypi-bin
- libraspberrypi-dev
- libraspberrypi-doc
- libraspberrypi0
- raspberrypi-bootloader
- raspberrypi-kernel
- raspberrypi-sys-mods
- raspi-config
- rpi-eeprom
Just by updating those packages, it is already automatically reflected in performance. The update was done with the 5 packages simultaneously, not one by one.
Well, this is super strange.
I've updated all packages one by one but the issue didn't occur or at least I didnt notice it. Then I've overclocked the Pi to 2GHz with an overvoltage of 6 and I noticed the random drops.
The Pi didn't throttle, whatsoever.
@jorensanbar Is your Pi overclocked? If yes, try running at stock speed
Edit: But I'm still seeing performance issues when downloading files from the samba server. So the read speed is affected even with no overclock.
So it might indeed be one of those packages @jorensanbar mentioned
@iandk , negative, my Pi is Stock
I'm going to reinstall the Pi Os Stock image again, and do the test skipping the SYS-Mod update. I don't know what other option I have.
Are you seeing those issues with both write and read speed? At least at the moment the write speed to the Pi seems fine, but read speed is much lower and quite unstable.
I'll also reinstall the Pi again and test if one of the packages you mentioned might be the problem. Well it must be, since before those recent updates everything was working just fine.
@popcornmix @pelwell
Does anyone of you know if any of the recent updates changed some behavior that could explain this performance drop?
The xbmc issue @popcornmix linked to ended with this patch appearing in the upstream kernel: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.10.15&id=8e081627f3a7f733a4955ee40b385c972f010f05 which has only today appeared in rpi-5.10.y. Other than that I'm not aware of anything, so I've been watching the bisecting with interest.
P.S. The latest rpi-update kernel is now on 5.10.16, so might be worth a try.
Thank you, how can I update to 5.10.16? With the rpi-update command? My kernel version is 5.10.11-v7l+
At least the read performance issues seem to be somehow related to the kernel, I've done a clean reinstall and tested the read/write performance. Write speed seems to be fine so far (at least without OC) even after updating/ upgrading all packages and the Kernel.
Read performance started to drop with the Kernel upgrade.
sudo rpi-update should do the trick. We normally warn at this point to back up your data first...
Yep, I've just updated to the latest kernel. Write @stock speed is fine so far, but the read performance still varies and is lower than with the old 5.4 kernel.