raspberry-pi-pcie-devices
raspberry-pi-pcie-devices copied to clipboard
geerlingguy
Test PCIe switches and adapters
I need to add a new section to the site for bridges/switches/adapters... because I'm finding that at least one of the three I've purchased doesn't seem to work with the Pi at all, giving me pcie: link is down on any attempt to use it with any PCIe device (including known good ones that work when I plug them in directly or through other adapters.
Right now I'm testing these:
IO Crest 1 to 2 ports PCIe x1 adapter — seems to work fine at least for some cards. Build quality of included floppy to molex power adapter is pretty bad, though:
Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip, which uses a 4-pin molex connector for power. This one seems like it came out of a video server of some sort. Got it on eBay:
PCIe 16x to 1x adapter (generic) - Works with everything so far, as if I plugged it straight in. It should though, since it's just a dumb extension cable:
Rosewill RCRC-17001 Mining Card 1x to 16x - Sometimes works, sometimes causes a kernel panic for some reason. I even bought two revisions (003, with a PCIe power connector, and 888, with a Molex 4-pin power connector), and both of them exhibit this weird behavior:
PCE164P-N06 VER008S PCIe 1x to 16x riser - Haven't tested yet, but I like how it has a plethora of power options!
One common annoyance is they all have different power adapter cables. I should probably just go buy a bulky PC power supply and drop it on my desk. I did by a 600W fully modular PSU.
With the 1 to 2 ports adapter, I'm able to get multiple devices to load, so that's nice:
$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
I'm going to confirm I can only transfer data at up to 3.2 Gbps though. Maybe less with the overhead of the PCI bridge?
To test NVMe + SATA, or anything plus anything requiring extra power... I'm going to need to grab an external PSU. I just don't have enough adapter cables to get from my single 4-pin molex power adapter to both the floppy connector on the PCIe switch plus SATA power for the drives.
Might as well buy a PC case at some point to contain all this crazy mess haha.
Anyways, I have my eye on the Redragon GC-PS003 600W fully modular PSU... mostly because it's close, relatively cheap, and available whenever. But I'm doing plenty of other stuff so it's not yet a priority.
Edit: Just ordered that PSU, and I'll have a little more fun with the switch next week!
The PSU is quite girthy, and also has to have a jumper across two pins on the motherboard connector so it will switch on. That was unforeseen, but obvious in hindsight.
I found another one possibly worth testing: https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-GN/353247990975
@Chaz6 - I'm actually going to try another switch, which might also be nice on the CM4, though it will likely cause a bit of strain on the connector... https://www.ebay.com/itm/PCIe-1x-to-Dual-PCIe-1x-90-Degree-Riser-Card-with-Power-PCIe-Switch-Expansion/273855896755?hash=item3fc3155cb3:g:fVwAAOSwHu5c4uvd
@Chaz6 I have that card (or a variant of it, anyway. Same design). The sticker covers it, but the chip on the board is of the same series as the IOCrest card that Jeff already tested, albeit one with a slightly expanded feature set (adding an extra port). They utilize the same kernel driver, so should ideally function exactly the same.
That said, I'm going to test it when my CM4 turns up one of these days and report back the results.
I'm currently testing that card from @Chaz6 in my PC (CM4 not here yet), but I'm having trouble getting multi-lane cards to work. Have you been experiencing the same? The only 1x card I have is a Realtek Gigabit card, which seems to work fine. But I haven't gotten any other card to work, I tested a Samsung 970 EVO (with 4x adapter), Mellanox 2 10GbE card and a dual gigabit card pulled from a HP server, none working. I cut open the connector to plug in longer cards since I don't have a riser card right now, but the 1x card still works in that port (so I guess nothing broke).
The cards not working do light up or get warm, but nothing in lspci, and nothing obvious in dmesg.
@markus-k - I can't speak to @Chaz6's card, but the IO Crest seems to so far only work with devices plugged in through slot 2 (maybe slot 1 is dead, maybe something in the Pi's PCIe stack doesn't work with it)... but I haven't done any rigorous testing with it yet.
I'm still focused right now on a number of individual cards and haven't yet done thorough testing with multiple cards.
Just to give an update, I figured out what the problem was (or is) after some digging into the switch and PCIe documentation. Not really related to the CM4, but probably useful to everyone thinking about cutting up their 1x PCIe switch ports to plug many multi-lane cards into their CM4.
When you just google about cutting open 1x PCIe ports everyone will tell you, it'll work no problem. That is not entirely true though, depending on how presence detection is implemented on the PCIe carrier board. PCIe cards have two pins, PRSNT1# and PRSNT2#, which need to be directly connected on the card. PCIe cards with more than one lane have multiple PRSNT2# pins, one for each lane (i.e. one for 1x, 4x, 8x, 16x), but only the one at the maximum number of supported lanes has to be connected to PRSNT1#.
Now, when you cut open your 1x port, PRSNT1# is connected to a pin that is not even present on the slot connector, and the 1x PRSNT2# is floating since the card supports more lanes. Thus, the PCIe switch does not detect that a card is plugged in and disables the port. When you use a proper 1x to Nx adapter, the correct PRSNTx# pins will usually be connected and the card will be detected.
Presence detection via PRSNTx# can be disabled in the switch, but the EEPROM is not fitted on the board, and you'd need to modify it's contents even if it was there. One other whacky solution, if you don't want to order a proper raiser, I used is to get out the soldering iron and botch a wire between PRSNT1# and PRSNT2# on the board, so the switch thinks there is always a card plugged in. PRSNT1# is connected to ground on this particular board, and PRSNT2# connected to the PCIe switch. I wouldn't really recommend doing this, but if you are too cheap to order yet an other PCIe adapter, it's probably fine.
And see, now cards with more than one lane are detected on the host.
@Chaz6 I have an almost identical 3 way card https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-L-/154052637457. Works fine with all the cards I've plugged into it so far (providing they work without the switch of course)
I also wanted to post an update—earlier I do remember having at least the Marvell 9215 (IO Crest) SATA card and Samsung 970 EVO NVMe working at the same time on my 2-slot switch, so it did work at least. I'll have to continue doing more testing. I have another new riser that has two slots and a different chip.
This one seems to be working too: https://www.aliexpress.com/item/32960541989.html
@danielkucera - Can you confirm having more than one device plugged in works, identifying all the devices? I noticed that many times it would only show devices plugged into the 2nd slot on my switch.
Can you try one of these PCIe to PCI(-X) cards too? Like this for example.
It would make it possible to use many old PCI cards on the RPi
@darkstar Tried already - https://www.raspberrypi.org/forums/viewtopic.php?p=1772997#p1772997 Results not positive as most cards seem to rely on mapping as I/O registers as that was the way that life used to work on x86 systems.
is anyone able to test ASM1184e chipset, I'm planing to build my own riser card with 1 x1 to 2 x4 + 2 M.2
It uses ASMedia chip ASM1184e: https://www.aliexpress.com/item/32868079091.html Used to be popular in mining community.
I have just put two cards in the first one pictured in Jeff's post and get this:
root@raspberrypi:~# lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
root@raspberrypi:~# dmesg | grep igb
[ 3.616121] igb: Intel(R) Gigabit Ethernet Network Driver
[ 3.616139] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.616786] igb 0000:04:00.0: enabling device (0000 -> 0002)
[ 3.842546] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 3.842572] igb 0000:04:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:ae
[ 3.842676] igb 0000:04:00.0: eth1: PBA No: E43709-006
[ 3.842694] igb 0000:04:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 3.843153] igb 0000:04:00.1: enabling device (0000 -> 0002)
[ 4.035061] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 4.035087] igb 0000:04:00.1: eth2: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:af
[ 4.035192] igb 0000:04:00.1: eth2: PBA No: E43709-006
[ 4.035208] igb 0000:04:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device.
I have tried the one dlednik recommended, this is the one for us guys!!
On my Amazon it is listed as a different brand but looks like its the same as is the case with many of these devices:
https://www.amazon.co.uk/gp/product/B07NZ5W8KJ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
To test it, I plugged in four devices, all four worked and the cool thing is that this is a mixture of x1 and x4 devices
All four devices working. A 4x Intel 10G network card, a 1x 2 port network card, a marvel 1x 4 port sata card and a 4x nvme card
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
05:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
pi@raspberrypi:~ $ dmesg | grep ixgbe
[ 3.517035] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 3.517054] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 3.522232] ixgbe 0000:03:00.0: enabling device (0000 -> 0002)
[ 4.738177] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 4.738550] ixgbe 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 4.738912] ixgbe 0000:03:00.0: MAC: 2, PHY: 1, PBA No: G18786-004
[ 4.738926] ixgbe 0000:03:00.0: 90:e2:ba:84:5e:f8
[ 4.746219] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[ 4.746761] libphy: ixgbe-mdio: probed
[ 4.747202] ixgbe 0000:03:00.1: enabling device (0000 -> 0002)
[ 5.932951] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 5.933290] ixgbe 0000:03:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 5.933631] ixgbe 0000:03:00.1: MAC: 2, PHY: 1, PBA No: G18786-004
[ 5.933637] ixgbe 0000:03:00.1: 90:e2:ba:84:5e:f9
[ 5.937171] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[ 5.937435] libphy: ixgbe-mdio: probed
pi@raspberrypi:~ $ dmesg | grep igb
[ 3.572119] igb: Intel(R) Gigabit Ethernet Network Driver
[ 3.572136] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.572567] igb 0000:06:00.0: enabling device (0000 -> 0002)
[ 3.760881] igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 3.760906] igb 0000:06:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:ae
[ 3.760999] igb 0000:06:00.0: eth1: PBA No: E43709-006
[ 3.761015] igb 0000:06:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 3.761513] igb 0000:06:00.1: enabling device (0000 -> 0002)
[ 3.946915] igb 0000:06:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 3.946940] igb 0000:06:00.1: eth2: (PCIe:2.5Gb/s:Width x1) 80:61:5f:05:0b:af
[ 3.947033] igb 0000:06:00.1: eth2: PBA No: E43709-006
[ 3.947049] igb 0000:06:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
pi@raspberrypi:~ $ dmesg | grep nvme
[ 3.463076] nvme nvme0: pci function 0000:04:00.0
[ 3.463299] nvme 0000:04:00.0: enabling device (0000 -> 0002)
[ 3.497918] nvme nvme0: allocated 32 MiB host memory buffer.
[ 3.506935] nvme nvme0: 4/0/0 default/read/poll queues
[ 3.516354] nvme0n1: p1
pi@raspberrypi:~ $ dmesg | grep SATA
[ 3.604608] ahci 0000:05:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[ 3.650437] ata1: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540100 irq 61
[ 3.650455] ata2: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540180 irq 61
[ 3.650471] ata3: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540200 irq 61
[ 3.650486] ata4: SATA max UDMA/133 abar m2048@0x601540000 port 0x601540280 irq 61
[ 4.123960] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 4.128017] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 4.128100] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 4.131946] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Happy days :D
I have just put two cards in the first one pictured in Jeff's post and get this:
[...]
What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device.
@PBXForums Have a look at my previous comment: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/14#issuecomment-735793184. Check if your Riser properly connects both PRSNT# pins.
@markus-k They work in the one in my above post, the two port one is now being scrapped in favor of that one which seems to be working perfectly.
Some devices (like the CM4IO) don't check presence with the PRSNT# pins but whether theres a 50 Ohm load on the PCIe lanes. You can achieve the same effect on the Pericom switches by just shorting the PRSNT# pins (or configuring the switch, which is a lot harder to do).
Testing this again with the IOCrest switch but with my 700W power supply for a NAS build... I finally seem to have it powered stably enough to get two devices working, and the system doesn't completely lock up when I run lspci:
$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
03:00.0 SATA controller: JMicron Technology Corp. Device 0585
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8125 (rev 04)
I was trying the same thing with the Exacq switch, and kept either having only one of the two cards work, or the Pi would lock up at various points through the boot process.
Note that I tried powering it both via dedicated 2A molex power supply and via my 700W (100+W to the 5v bus) PSU, and it seemed to work more stably connected through my PSU. It still required one extra reboot before both cards were fully functional though.
Seems to be extremely stable now (the IOCrest switch). I think for any of these switches, a solid power supply capable of handling 3-5A or more is going to be pretty important.
But even then, some of the adapters and switches just... fail sometimes, and cause kernel panic. No idea why that happens randomly.
Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only?
My goal is to connect a drive and a VPU (like the coral).
In doing that, would it be better to just leave the drive on USB?
If this isn't the place to ask, feel free to delete or point me in the right direction to learn more.
Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only?
They're using USB3 cables purely as a convenient physical cable. They do not convert to/from USB at an electrical or protocol level. Connecting to a genuine USB host or slave may well result in damage to either or both ends.
As Jeff noted, there is also no standard wiring with these adapters, so ensure you use the correct pairs of adapters.
I've had my eye on these for my mining shinanigans and they look perfect for Raspberry Pi usage given their connectivity is a USB socket instead of a PCIe slot but can't seem to find them anywhere else and the vendor didn't answer my request for quote but thought I'd share anyway incase anyone knows where to get them.
https://www.alibaba.com/product-detail/pcie-splitter-pcie-to-pci-adapter_60572718142.html
Was anyone able to test the PI7C9X2G404SL chip? I just need a working PCIe switch that works with the CM4, Ubuntu Core 20. Unless I have to look at the ASM1184e PCIe switch variants?
Was anyone able to test the PI7C9X2G404SL chip? I just need a working PCIe switch that works with the CM4, Ubuntu Core 20.
I tried that chip in Ubuntu Core 20 without success -- but I haven't confirmed the adapter card definitely works. Interested to know experience of other people. -- not had chance to look to see if it was a driver issue for ubuntu
