firmware
firmware copied to clipboard
raspberrypi
Can't boot Pi 5 via NVMe behind PCIe switch / bridge
Describe the bug
I am unable to boot a Raspberry Pi 5 from an external NVMe SSD if used behind a PCIe switch (e.g. not as the root device on the external connection).
To reproduce
- Ensure you have an NVMe SSD that can boot the Raspberry Pi 5 when connected directly. (
BOOT_ORDER=0xf25416) - Connect a PCI Express switch to the external PCIe connection on the Raspberry Pi 5.
- Move the NVMe SSD you previously used to boot the Pi 5 to a port behind the PCIe switch.
- Attempt to boot the Raspberry Pi 5 off the NVMe SSD.
Expected behaviour
I would expect the NVMe SSD to be selected for boot, wherever it is enumerated on the PCIe bus.
Actual behaviour
The Raspberry Pi 5 bootloader attempts to load nvme but fails, likely due to it only enumerating devices directly attached to the external port, and not walking down the tree of any other connected PCIe bridges...
System Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:
- Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
- Which OS and version (
cat /etc/rpi-issue)? - Which firmware version (
vcgencmd version)? - Which kernel version (
uname -a)?
Logs
Click to expand full bootloader log (captured via UART)
RPi: BOOTSYS release VERSION:9d494316 DATE: 2023/09/13 TIME: 11:37:06
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1694601426 serial e3ae743b boardrev d04170 stc 1604502
AON_RESET: 00000003 PM_RSTS 00001000
RP1_BOOT chip ID: 0x20001927
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
PMIC reset-event 00000000 rtc 65299da7 alarm 00000000 enabled 0
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267
DDR 4267 1 0 64 152
RP1_BOOT chip ID: 0x20001927
RP1_BOOT chip ID: 0x20001927
RP1_BOOT: fw size 25968
PCI2 init
PCI2 reset
PCIe scan 00001de4:00000001
RP1_CHIP_INFO 20001927
RPi: BOOTLOADER release VERSION:9d494316 DATE: 2023/09/13 TIME: 11:37:06
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1694601426 serial e3ae743b boardrev d04170 stc 4469863
AON_RESET: 00000003 PM_RSTS 00001000
status
USB_PD CONFIG 0 43
Boot mode: NVME (06) order f41
PCI1 init
PCI1 reset
USB-PD: src-cap PDO object1 0x0a0191f4
Current 5000 mA
Voltage 5000 mV
USB-PD: src-cap PDO object2 0x0002d12c
Current 3000 mA
Voltage 9000 mV
USB-PD: src-cap PDO object3 0x0003c0e1
Current 2250 mA
Voltage 12000 mV
USB-PD: src-cap PDO object4 0x0004b0b4
Current 1800 mA
Voltage 15000 mV
Failed to open device: 'nvme'
Retry NVME 1
Failed to open device: 'nvme'
Boot mode: SD (01) order f4
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
EMMC
SD retry 1 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD retry 2 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SDV1
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Retry SD 1
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
EMMC
SD retry 1 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD retry 2 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SDV1
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Boot mode: USB-MSD (04) order f
XHCI-STOP
xHC0 ver: 0 HCS: 00000085 00000085 00000085 HCC: 00000085
USBSTS 85
xHC0 ver: 0 HCS: 00000085 00000085 00000085 HCC: 00000085
xHC0 ports 0 slots 133 intrs 0
USB xHC init failed
Boot mode: RESTART (0f) order 0
Restart 0 max -1
Boot mode: NVME (06) order f41
PCI1 init
PCI1 reset
Failed to open device: 'nvme'
Retry NVME 1
Failed to open device: 'nvme'
Additional context
When behind the bridge, here is the hierarchy according to lspci:
pi@pi5:~ $ lspci -tv
-+-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-01.0-[03]--
| \-02.0-[04]----00.0 KIOXIA Corporation Device 0010
\-[0001:00]---00.0-[01]----00.0 Device 1de4:0001
I know for the Compute Module 4, the concern was a lack of space in the bootloader to successfully enumerate all PCIe devices, no matter where they are on the bus. Does the Pi 5's bootloader overcome that limitation? I don't expect this to work on launch day, but it is something I think a lot of people would like to do (e.g. stack an 'NVMe + 2.5G Ethernet' HAT, or 'NVMe + WiFi 7' HAT, etc. on top).
@peterharperuk - I'm testing with this I/O Crest switch:
pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
0000:04:00.0 Non-Volatile memory controller: KIOXIA Corporation Device 0010 (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
I've just come across a HAT+ (Geekworm X1004) with an ASM1182e PCIe switch for two SSDs and the manufacturer claims it can't be used for booting at this time. It looks related.
@cnxsoft - It is most definitely related! And thanks for you post on that board, I really do hope Raspberry Pi can support at least a few switches (the asmedia ones seem extremely popular for smaller companies. I don't see Pericom that often).
I heard the guys from Pineberry are also already experimenting with a hat that supports 2 m.2 devices... (no promises though.. ) 2 NVME drives or 1 NVME drive and room for a pcie coral TPU would be nice but of course we need to be able to boot from it too if a pcie switch is involved :)
Removed off topic comment about a CM4. This issue is specifically about Pi5 / BCM2712
That's covered in the documentation. From https://www.raspberrypi.com/documentation/computers/raspberry-pi-5.html#pcie-gen-3-0:
The Raspberry Pi 5 is not certified for Gen 3.0 speeds, and connections to PCIe devices at these speeds may be unstable.
If Gen 3 was guaranteed to be stable then it would be enabled by default.
@Ronald1817 - The title of this GitHub issue is "Can't boot Pi 5 via NVMe behind PCIe switch / bridge" and is meant to discuss that topic, not general NVMe SSD issues you may be having. The Raspberry Pi forums, or the manufacturer of the HAT / Bottom you're using, would be a better avenue for questions about Pi boot issues with NVMe.
That is all that is implied by @timg236—this GitHub repository is for issues pertaining to the Pi firmware, and this issue is about a specific feature request. These comments don't have anything to do with that, so typically the maintainers will clean them out at some point.
A new dual NVMe rpi5 product has appeared with a PCIe switch in it https://pimoroni.com/nvmeduo Anyone know what the switch part number is?
@greyltc - Looks like ASM1182e. Same as the Geekworm—and Pineberry Pi!
@timg236 / @pelwell — it seems like by some stroke of luck, all the main vendors of PCIe addon boards have settled on the asmedia ASM1182e as their PCIe gen 2 switch of choice... would it be possible for that chip (at least for now) to be supported for switch traversal for NVMe boot?
https://pineberrypi.com/products/hatdrive-ai-coral-edge-tpu-bundle-nvme-2230-2242-gen-2-for-raspberry-pi-5 This new very sweet toy is also going to use the ASMedia switch, at this point I guess it's going to be the "standard" for most devices.
Recently I have bought an PCIe HAT from aliexpress.com. The link is https://www.aliexpress.us/item/3256806347359812.html. This has one 2.5Gb Eth and one NVM. This board has the 'ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch'. I have installed an NVMe drive of 512G from my old laptop. I had issues booting it from. Then switched back to SD card boot using EEPROM recovery image.
But I found two interesting things:
- First, I was able to use NVMe for OS
- Second, I was able to fully boot from NVMe (without setting the boot order to NVMe but with the help of SD card)
Let me explain below what I did, I didn't go in depth to find out how it worked. But at a high level,
- To make bootable NVMe, instead of installing from the scratch in the NVMe, I just did a 'dd' from SD card to NVMe (This copied the UUIDs as well)
- Then changed the boot order to NVMe and removed the SD card
- First many attempts to boot from NVMe without SD card didn't work.
- Next, I have updated the boot order back to SD card and used my original SD card (which was used to duplicate to NVMe)
- Then it booted, I thought it booted from SD card. But when I checked, the mounted partitions, I saw, the /boot/firmware was from SD card and the '/' (root) partition was from my NVMe. So basically, the OS was running on NVMe.
I didn't know what I did except expanding the root partition size to utilize the full NVMe (512G) size, (SD card was 128G), I resized the partition in NVMe. While preparing to answer this, I found that the below in lsblk command.
mmcblk0 179:0 0 116.2G 0 disk ├─mmcblk0p1 179:1 0 512M 0 part └─mmcblk0p2 179:2 0 115.7G 0 part nvme0n1 259:0 0 476.9G 0 disk ├─nvme0n1p1 259:1 0 512M 0 part /boot/firmware └─nvme0n1p2 259:2 0 476.4G 0 part /
Both boot and root partitions are used from NVMe only. Now this is running as my home router / gateway. So, I can't do much tests immediately. But I will try to reboot multiple times/ try without SD card and see what's happening when I get a chance and post it here.
Can I please add the ASM1184e PCIe packet switch chip to this issue? It shows up on the ZS ZHISHANG PCI-E X1 to 4 PCI-E X16 Expansion Riser as documented by @geerlingguy.
The ASM1182e is mentioned above. It's a 2 by PCIe chip vs the ASM1184e which is 4 by PCIe chip; so similar(ish).
Boards available on Amazon via:
- https://www.amazon.com/dp/B075DF9L5V (available)
- https://www.amazon.com/dp/B07DDFXV99 (presently unavailable) - the one I purchased.
- https://www.amazon.com/dp/B07KQ4D7ZS (available)
I will note that this PCIe bridge works fine once the system is booted.
@greyltc - Looks like
ASM1182e. Same as the Geekworm—and Pineberry Pi!@timg236 / @pelwell — it seems like by some stroke of luck, all the main vendors of PCIe addon boards have settled on the asmedia
ASM1182eas their PCIe gen 2 switch of choice... would it be possible for that chip (at least for now) to be supported for switch traversal for NVMe boot?
In principle any switch that follows the spec should work with generic firmware (once written). Supporting / debugging individual switch/PCIe devices would be up to the HAT designer i.e. debugging low level electronic issues.
To be clear, my obeys the spec I mean that anything that requires a ton of quirks an interop fixes just won't be supported. No immediate plans to do this though
Go easy with the adverts...
Wasn’t trying to plug anything, just geeking out over the tech specs a bit too much, maybe. There are no other solutions like this so it was almost impossible to keep the name of the board out of the discussion. Thanks for the nudge!
Going back to the topic. If someone desperately needs to boot from NVMe/SATA behind the switch it works with both the UEFI approach and uboot(it applies to the CM4 too). Having to place a secondary bootloader on the SD card is not perfect but no changes are required to the OS itself and this is a huge plus.
Agreed. The intention would be to support booting from an NVMe driver behind a switch, probably only simple hardware topologies would be supported
@timg236 - The other use case I have is to have multiple boot drives in a compact solution, so a Pi could boot into different OSes more easily than with a bunch of USB stuff hanging off it (or PXE boot). That... would be interesting to do from the Pi OS default firmware experience, though a UEFI bootloader could have boot order priority.
I don't know the default way the bootloader picks which device to boot from first, if there are multiple on one bus like USB. Is it by device ID?
The bootloader enumerates MSD in parallel and you get the first one that looks like valid OS. It’s mentioned in the msd docs. The design goal is to do the right thing based on hardware detection and sensible defaults without ever inflicting a PC BIOS style UI on the end user. That said there’s scope for advanced users to have more flexibility eg via boot loader config specified GUIDs or ids. It very much a second order feature
This issue is to track the development of native support for PCIe switches in the Raspberry Pi5 bootloader. When this is released we'll tag this bug so feel free to subscribe to this to get notified about early software release ... which will be via rpi-update.
I've deleted all the off topic comments about general NVMe support / booting from from EMMC first.
The best place to discuss interop / workarounds or things that aren't directly related to the software change is on the Raspberry Pi forums e.g. Advanced Users.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 238.5G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part /boot/firmware
└─nvme0n1p2 259:2 0 238G 0 part /
nvme1n1 259:3 0 232.9G 0 disk
├─nvme1n1p1 259:4 0 512M 0 part
└─nvme1n1p2 259:5 0 232.4G 0 part
I have just installed a Pimoroni NVMe Base Duo w/two 500GB disks (ASM1182e). I updated the firmware to version Sat 20 Apr 2024 and then used the HTTP installer with no SD Card or USB disk installed. This worked just fine and installed the system on the NVMe disk of choice. Of course, this won't boot directly at present. I then prepared a USB stick using RPI-Imager, then in /boot/firmware/cmdline.txt I simply replaced the disk ID for the ROOT=PARTUUID=xxxxxxxxx- stanza with the partuuid value from the cmdline.txt file on the NVMe drive. This leads to the lsblk situation depicted by @peterharperuk. above.
The USB stick just serves to circumvent the bootloader's inability to boot in this scenario. But then, everything runs as if it were booted from NVMe. Once booted, the USB key can be removed to save power - as long as nobody tries to reboot. I can even set the boot order to 0xf46 (NVMe first, then USB, then loop).
Of course, I have to manually copy updates to the /boot/firmware partition from NVMe to the USB Key later on. For me, this appears the most simple way to have a system installed on NVMe at this time.
Can I use an SD card instead of a bootable USB?
But hoooow? And how long until I can play with this :D
@timg236 ...because somewhat off-topic here (just a workaround), I deleted my post here and moved it to https://forums.raspberrypi.com/viewtopic.php?t=369796
