linux icon indicating copy to clipboard operation
linux copied to clipboard
verified publisher icon raspberrypi

Performance Degradation and I/O Errors with Btrfs RAID, Multiple USB-to-SATA Disks on Raspberry Pi 5

Open Boma123 opened this issue 1 year ago • 4 comments

Describe the bug

I'm encountering severe performance degradation and I/O errors when connecting multiple SATA drives via USB-to-SATA adapters on the Raspberry Pi 5. When three drives are connected simultaneously to the Pi's USB 3.0 ports, file transfer speeds drop below USB 2.0 levels (~20-22 MB/s) and I/O errors, disconnections, and resets are observed in dmesg. These issues occur despite confirming stable power delivery to the external drives. The problem persists regardless of UAS being enabled or disabled (switched to usb-storage via quirks).

Setup:

  • Raspberry Pi 5 (4GB) running Debian Bookworm (headless).
  • Three 3.5" HDDs: 1x WD Caviar Green (1.5 TB), 2x WD RED NAS (3TB each, configured in a Btrfs RAID 1 array)
  • USB-to-SATA adapters: Generic adapters from Aliexpress (chipset unknown), same model for all three disks
  • External 12V power supply for the drives, verified stable under load with a voltmeter.
  • M.2 hat with NVMe drive for OS

Behavior: When all three drives are connected to USB 3.0 (some through a USB 3.0 hub), file transfer speeds drop to 20-22 MB/s, and the following errors appear in dmesg:

  • UAS-related aborts and resets.
  • Device disconnects.
  • I/O errors and file system corruption.

When only the RAID or the single drive are connected, read/write performance is as expected; no errors or performance degradation occurs in this case. The USB 3.0 hub used in the setup (UGREEN USB 3.0 Hub 4 Ports) works perfectly fine (read/write speeds >110MB/s) when only two drives are connected to it.

Workaround: Connecting the WD Caviar Green drive via a USB 2.0 port avoids the disconnection issues but now I am limited by USB 2 connection speeds (effective ~33 MB/s). No errors appear in dmesg under this configuration.

Solution attempts / tests:

  • External disk power is stable. No voltage drops under load (disks spinup).
  • USB3 port combinations: changing between all three drives connected thru hub or just two thru hub and plugging one into the remaining USB3 port on PI has no effect.
  • Disabled UAS and switched to usb-storage: No improvement. Errors still appear in dmesg, including device disconnections
  • Changed I/O scheduler from mq-deadline to bfq: No observable difference in performance or error behavior.
  • System I/O and memory usage: Monitored CPU and I/O, no significant CPU load, and no signs of memory exhaustion.

Steps to reproduce the behaviour

Set up a Btrfs RAID 1 array with two SATA hard drives connected via USB 3.0 (using USB-to-SATA adapters and a USB3 hub). Connect a third SATA drive via USB 3.0 (thru hub). Begin transferring large files (1GB+) between the RAID array and the third drive. Observe performance degradation (speeds dropping to 20-22 MB/s) and dmesg logs for I/O errors, USB disconnects, and device resets.

Device (s)

Raspberry Pi 5

System

https://pastebin.com/3XB1Xasm

Logs

dmesg sample entry when all 3 disks are connected via usb 3: [ 11.940773] EXT4-fs (sdc): mounted filesystem 061a8eb4-7dd0-4d38-a03c-d71ff5fef21a r/w with ordered data mode. Quota mode: none. [ 25.621174] systemd[920]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set [ 125.886634] sd 2:0:0:0: [sdc] tag#19 uas_eh_abort_handler 0 uas-tag 2 inflight: IN [ 125.886644] sd 2:0:0:0: [sdc] tag#19 CDB: opcode=0x28 28 00 43 e9 40 00 00 02 00 00 [ 128.190107] sd 2:0:0:0: [sdc] tag#17 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT [ 128.190112] sd 2:0:0:0: [sdc] tag#17 CDB: opcode=0x2a 2a 00 57 44 00 00 00 00 08 00 [ 128.206107] scsi host2: uas_eh_device_reset_handler start [ 133.537835] xhci-hcd xhci-hcd.1: Timeout while waiting for setup device command [ 133.745917] usb 4-1: reset SuperSpeed USB device number 2 using xhci-hcd [ 133.770602] scsi host2: uas_eh_device_reset_handler success [ 133.770721] sd 2:0:0:0: [sdc] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=38s [ 133.770725] sd 2:0:0:0: [sdc] tag#19 CDB: opcode=0x28 28 00 43 e9 40 00 00 02 00 00 [ 133.770728] I/O error, dev sdc, sector 1139359744 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2

Additional context

No response

Boma123 avatar Oct 23 '24 06:10 Boma123

Turns out, that I get sudden USB disconnects even when connecting the third drive to a USB2 port. Problem seems intermittent. Error below happened while rsyncing a large directory.

[ 18.491061] systemd[921]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set [ 438.976068] usb 3-2: USB disconnect, device number 2 [ 438.976293] sd 2:0:0:0: [sdc] tag#11 uas_zap_pending 0 uas-tag 1 inflight: CMD [ 438.976299] sd 2:0:0:0: [sdc] tag#11 CDB: opcode=0x2a 2a 00 55 2f 28 00 00 04 00 00 [ 438.976305] sd 2:0:0:0: [sdc] tag#8 uas_zap_pending 0 uas-tag 2 inflight: CMD [ 438.976308] sd 2:0:0:0: [sdc] tag#8 CDB: opcode=0x2a 2a 00 55 2f 2c 00 00 04 00 00 [ 438.976316] sd 2:0:0:0: [sdc] tag#9 uas_zap_pending 0 uas-tag 3 inflight: CMD [ 438.976317] sd 2:0:0:0: [sdc] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s [ 438.976319] sd 2:0:0:0: [sdc] tag#9 CDB: opcode=0x2a 2a 00 55 2f 30 00 00 04 00 00 [ 438.976326] sd 2:0:0:0: [sdc] tag#10 uas_zap_pending 0 uas-tag 4 inflight: CMD [ 438.976327] sd 2:0:0:0: [sdc] tag#11 CDB: opcode=0x2a 2a 00 55 2f 28 00 00 04 00 00 [ 438.976330] sd 2:0:0:0: [sdc] tag#10 CDB: opcode=0x2a 2a 00 55 2f 34 00 00 04 00 00 [ 438.976331] I/O error, dev sdc, sector 1429153792 op 0x1:(WRITE) flags 0x4000 phys_seg 32 prio class 2 ... [ 438.976490] EXT4-fs warning (device sdc): ext4_end_bio:343: I/O error 10 writing to inode 76940386 starting block 178646912) [ 438.976492] sd 2:0:0:0: [sdc] tag#3 CDB: opcode=0x2a 2a 00 55 2f 20 00 00 04 00 00 [ 438.976495] sd 2:0:0:0: [sdc] tag#7 uas_zap_pending 0 uas-tag 30 inflight: CMD [ 438.976497] sd 2:0:0:0: [sdc] tag#7 CDB: opcode=0x2a 2a 00 55 2f 24 00 00 04 00 00 [ 438.976497] Buffer I/O error on device sdc, logical block 178644992 [ 438.976500] Buffer I/O error on device sdc, logical block 178644993 [ 438.976502] Buffer I/O error on device sdc, logical block 178644994 [ 438.976504] Buffer I/O error on device sdc, logical block 178644995 [ 438.976510] Buffer I/O error on device sdc, logical block 178644996 [ 438.976512] Buffer I/O error on device sdc, logical block 178644997 [ 438.976514] Buffer I/O error on device sdc, logical block 178644998 [ 438.976515] Buffer I/O error on device sdc, logical block 178644999 [ 438.976518] Buffer I/O error on device sdc, logical block 178645000 [ 438.976520] Buffer I/O error on device sdc, logical block 178645001 [ 438.976805] EXT4-fs warning (device sdc): ext4_end_bio:343: I/O error 17 writing to inode 76940390 starting block 178648960) [ 438.976858] EXT4-fs warning (device sdc): ext4_end_bio:343: I/O error 10 writing to inode 76940386 starting block 178644864) [ 438.977107] Aborting journal on device sdc-8. [ 438.977137] Buffer I/O error on dev sdc, logical block 183009280, lost sync page write [ 438.977256] EXT4-fs warning (device sdc): ext4_end_bio:343: I/O error 17 writing to inode 76940390 starting block 178637411) [ 438.977307] JBD2: I/O error when updating journal superblock for sdc-8. [ 438.977323] EXT4-fs error (device sdc): ext4_journal_check_start:84: comm kworker/u11:2: Detected aborted journal [ 438.987986] EXT4-fs warning (device sdc): ext4_end_bio:343: I/O error 17 writing to inode 76940394 starting block 178651008) [ 438.988686] JBD2: Detected IO errors while flushing file data on sdc-8 [ 438.988700] Buffer I/O error on dev sdc, logical block 0, lost sync page write [ 438.988712] EXT4-fs (sdc): I/O error while writing superblock [ 438.988715] EXT4-fs (sdc): Remounting filesystem read-only [ 438.994232] EXT4-fs (sdc): ext4_do_writepages: jbd2_start: 512 pages, ino 76940394; err -30 [ 439.003000] Buffer I/O error on dev sdc, logical block 1, lost async page write [ 439.003005] Buffer I/O error on dev sdc, logical block 2, lost async page write [ 439.003007] Buffer I/O error on dev sdc, logical block 3, lost async page write [ 439.003010] Buffer I/O error on dev sdc, logical block 4, lost async page write [ 439.003016] Buffer I/O error on dev sdc, logical block 25690122, lost async page write [ 439.003023] Buffer I/O error on dev sdc, logical block 26214414, lost async page write [ 439.003027] Buffer I/O error on dev sdc, logical block 38797314, lost async page write [ 439.003030] Buffer I/O error on dev sdc, logical block 41418760, lost async page write [ 439.181120] sd 2:0:0:0: [sdc] Synchronizing SCSI cache [ 439.413076] sd 2:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=DRIVER_OK [ 439.601948] EXT4-fs (sdc): unmounting filesystem 061a8eb4-7dd0-4d38-a03c-d71ff5fef21a. [ 446.141071] usb 3-2: new high-speed USB device number 3 using xhci-hcd [ 446.340505] usb 3-2: New USB device found, idVendor=7825, idProduct=a2a4, bcdDevice=41.01 [ 446.340510] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 446.340514] usb 3-2: Product: Best USB Device [ 446.340516] usb 3-2: Manufacturer: ULT-Best [ 446.340519] usb 3-2: SerialNumber: 235678C218CA [ 446.341850] scsi host2: uas [ 446.352347] scsi 2:0:0:0: Direct-Access WDC WD15 EADS-00R6B0 4101 PQ: 0 ANSI: 6 [ 446.354785] sd 2:0:0:0: Attached scsi generic sg2 type 0 [ 446.368532] sd 2:0:0:0: [sdc] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB) [ 446.368539] sd 2:0:0:0: [sdc] 4096-byte physical blocks [ 446.368705] sd 2:0:0:0: [sdc] Write Protect is off [ 446.368709] sd 2:0:0:0: [sdc] Mode Sense: 5f 00 00 08 [ 446.369034] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 446.369131] sd 2:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes [ 446.369134] sd 2:0:0:0: [sdc] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes) [ 446.372107] sd 2:0:0:0: [sdc] Attached SCSI disk

Boma123 avatar Oct 23 '24 12:10 Boma123

[ 438.976068] usb 3-2: USB disconnect, device number 2

This is likely a power supply issue. You said you had checked the 12V, but this is likely on the USB/5V power supply.

popcornmix avatar Oct 23 '24 12:10 popcornmix

@popcornmix thx, just came to the same conclusion when I saw that Pi5 can only supply a combined 1.6A across all USB ports. Have ordered a powered USB3 hub which should resolve the issue. Will report back

Boma123 avatar Oct 23 '24 12:10 Boma123

Beware cheap SATA adapters that claim to use 12V, but they cop out by drawing 5V from Vbus instead of regulating it internally, assuming that most of the current draw is from 12V (as is the case with HDDs). SSDs are usually 5V powered.

P33M avatar Oct 23 '24 17:10 P33M

I can confirm that this is a USB power draw problem. Switched to a powered USB hub and all three disks are performing as expected, no more random disconnects.

I did some additional measurements: each USB/SATA adapter is pulling about 1A@12V peak (during disk spin up) from the external power supply, and about 0.6A@5V peak (during file transfers) from the USB port. When all three adapters are active, the combined USB power draw exceeds the 1.6A max of the Pi5 - hence the disconnects.

Thanks for the hint. Closing this issue.

Boma123 avatar Nov 01 '24 14:11 Boma123

I did some additional measurements: each USB/SATA adapter is pulling about 1A@12V peak (during disk spin up) from the external power supply, and about 0.6A@5V peak (during file transfers) from the USB port. When all three adapters are active, the combined USB power draw exceeds the 1.6A max of the Pi5 - hence the disconnects.

You might imagine (hope) that a drive with an external 12V power supply would generate its own 5V supply from this if it needs to draw significant power, but sadly that is often not the case.

popcornmix avatar Nov 01 '24 14:11 popcornmix