raspberry-pi-pcie-devices
raspberry-pi-pcie-devices copied to clipboard
Published
20 hours ago •
geerlingguy
geerlingguy
Test Dell Intel 942V6 X520-DA2 Dual-Port 10Gb SFP+ PCIe NIC
This is a fun one, sent to me by Jacob Hiltz:
It's a Dell Intel 942V6 X520-DA2 Dual-Port 10Gb SFP+ PCIe NIC, and it's a little over a hundred bucks new. This was pulled from a Dell server and I have a couple SFP+ 10 Gbps cables (see related: #21).
From my experience with the ASUS 10G adapter (see #15), I know I'll probably not get more than about 3.2 Gbps through the card... but it would be very interesting to see if both 10G interfaces show up and could be used simultaneously.
Having better latency and a solid connection like SFP+ opens up some interesting possibilities. And it's just darn fun.
This should work just fine once the intel driver is compiled in:
root@raspberrypi:~# lspci 00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20) 01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
iperf -c 192.168.35.11
Client connecting to 192.168.35.11, TCP port 5001 TCP window size: 325 KByte (default)
[ 3] local 192.168.35.10 port 43468 connected with 192.168.35.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.69 GBytes 3.17 Gbits/sec
A little more with mtu 9000:
iperf -c 192.168.35.11
Client connecting to 192.168.35.11, TCP port 5001 TCP window size: 325 KByte (default)
[ 3] local 192.168.35.10 port 43576 connected with 192.168.35.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.90 GBytes 3.35 Gbits/sec
@PBXForums - Nice! It seems like you're right at that bus limit that I hit testing the ASUS card (3.36 Gbps).
Can confirm this works just fine with X520-DA1 and X520-DA2 cards. Using 64-bit raspbian on kernel 5.10.17-v8+, tested with reference design boards from Dell, HPE, SuperMicro, and Intel themselves; all work just fine, to nobody's real surprise.
If anyone would like to get it working themselves, since the raspbian default kernel doesn't build ixgbe and rebuilding the kernel takes untold centuries (or cross-compilation), here's a copy of the intel v5.11.3 driver source with a dkms.conf added - clone to /usr/src/ixgbe-5.11.3 and do the usual dkms install commands.
There's also a .deb package in the releases on that repo which should handle all the dkms setup for you.
If recompiling the kernel, though, in menuconfig select:
Device Drivers
> Network device support
> Ethernet driver support
> Intel(R) 10GbE PCI Express adapters support
Adds into .config:
CONFIG_ETHERNET=y
+CONFIG_MDIO=m
CONFIG_NET_VENDOR_3COM=y
# CONFIG_VORTEX is not set
# CONFIG_TYPHOON is not set
@@ -2441,7 +2442,9 @@ CONFIG_NET_VENDOR_INTEL=y
# CONFIG_IGB is not set
# CONFIG_IGBVF is not set
# CONFIG_IXGB is not set
-# CONFIG_IXGBE is not set
+CONFIG_IXGBE=m
+CONFIG_IXGBE_HWMON=y
+CONFIG_IXGBE_IPSEC=y
# CONFIG_IXGBEVF is not set
# CONFIG_I40E is not set
# CONFIG_I40EVF is not set
After a recompile, copy, then reboot, I am seeing:
pi@raspberrypi:~ $ sudo lspci -vvvv
...
01:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
Subsystem: Intel Corporation 10GbE 2P X520 Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 65
Region 0: Memory at 600000000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at <unassigned> [disabled]
Region 4: Memory at 600300000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 600200000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <8us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Vital Product Data
Product Name: X520 10GbE Controller
Read-only fields:
[PN] Part number: G73129
[MN] Manufacture ID: 31 30 32 38
[V0] Vendor specific: FFV18.0.0
[V1] Vendor specific: DSV1028VPDR.VER1.0
[V3] Vendor specific: DTINIC
[V4] Vendor specific: DCM10010081D521010081D5
[V5] Vendor specific: NPY2
[V6] Vendor specific: PMT12345678
[V7] Vendor specific: NMVIntel Corp
[RV] Reserved: checksum good, 0 byte(s) reserved
Read/write fields:
End
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number b4-96-91-ff-ff-08-9d-2c
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
VF offset: 384, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
01:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
Subsystem: Intel Corporation 10GbE 2P X520 Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 0
Region 0: Memory at 600100000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at <unassigned> [disabled]
Region 4: Memory at 600304000 (64-bit, non-prefetchable) [size=16K]
[virtual] Expansion ROM at 600280000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <8us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [e0] Vital Product Data
Product Name: X520 10GbE Controller
Read-only fields:
[PN] Part number: G73129
[MN] Manufacture ID: 31 30 32 38
[V0] Vendor specific: FFV18.0.0
[V1] Vendor specific: DSV1028VPDR.VER1.0
[V3] Vendor specific: DTINIC
[V4] Vendor specific: DCM10010081D521010081D5
[V5] Vendor specific: NPY2
[V6] Vendor specific: PMT12345678
[V7] Vendor specific: NMVIntel Corp
[RV] Reserved: checksum good, 0 byte(s) reserved
Read/write fields:
End
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number b4-96-91-ff-ff-08-9d-2c
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
VF offset: 384, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
And:
pi@raspberrypi:~ $ dmesg | grep pci
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=B8:27:EB:5C:89:43 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1 root=PARTUUID=38df951b-02 rootfstype=ext4 elevator=deadline fsck.repair=yes pci=pcie_bus_perf rootwait
[ 1.224473] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.224515] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.224601] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 1.224692] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
[ 1.271853] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[ 1.272234] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 1.272265] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.272293] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[ 1.272394] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 1.272642] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 1.276264] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[ 1.276567] pci 0000:01:00.0: [8086:154d] type 00 class 0x020000
[ 1.276644] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[ 1.276684] pci 0000:01:00.0: reg 0x18: [io 0x0000-0x001f]
[ 1.276751] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[ 1.276790] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[ 1.277031] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 1.277125] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 1.277444] pci 0000:01:00.1: [8086:154d] type 00 class 0x020000
[ 1.277517] pci 0000:01:00.1: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[ 1.277557] pci 0000:01:00.1: reg 0x18: [io 0x0000-0x001f]
[ 1.277623] pci 0000:01:00.1: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[ 1.277662] pci 0000:01:00.1: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[ 1.277934] pci 0000:01:00.1: PME# supported from D0 D3hot
[ 1.281339] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 1.281393] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6003fffff]
[ 1.281429] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x6000fffff 64bit]
[ 1.281482] pci 0000:01:00.1: BAR 0: assigned [mem 0x600100000-0x6001fffff 64bit]
[ 1.281532] pci 0000:01:00.0: BAR 6: assigned [mem 0x600200000-0x60027ffff pref]
[ 1.281561] pci 0000:01:00.1: BAR 6: assigned [mem 0x600280000-0x6002fffff pref]
[ 1.281591] pci 0000:01:00.0: BAR 4: assigned [mem 0x600300000-0x600303fff 64bit]
[ 1.281641] pci 0000:01:00.1: BAR 4: assigned [mem 0x600304000-0x600307fff 64bit]
[ 1.281689] pci 0000:01:00.0: BAR 2: no space for [io size 0x0020]
[ 1.281713] pci 0000:01:00.0: BAR 2: failed to assign [io size 0x0020]
[ 1.281813] pci 0000:01:00.1: BAR 2: no space for [io size 0x0020]
[ 1.281839] pci 0000:01:00.1: BAR 2: failed to assign [io size 0x0020]
[ 1.281866] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 1.281898] pci 0000:00:00.0: bridge window [mem 0x600000000-0x6003fffff]
[ 1.281957] pci 0000:00:00.0: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512
[ 1.282009] pci 0000:01:00.0: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512
[ 1.282061] pci 0000:01:00.1: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512
[ 3.932276] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 5.409265] ixgbe 0000:01:00.1: of_irq_parse_pci: failed with rc=-22
pi@raspberrypi:~ $ dmesg | grep ixgbe
[ 3.919119] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 3.919141] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 3.932332] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 5.398272] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 5.398610] ixgbe 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 5.398939] ixgbe 0000:01:00.0: MAC: 2, PHY: 1, PBA No: G73131-008
[ 5.398953] ixgbe 0000:01:00.0: b4:96:91:08:9d:2c
[ 5.408195] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[ 5.408761] libphy: ixgbe-mdio: probed
[ 5.409265] ixgbe 0000:01:00.1: of_irq_parse_pci: failed with rc=-22
[ 5.409315] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[ 6.614332] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 6.614666] ixgbe 0000:01:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 6.614995] ixgbe 0000:01:00.1: MAC: 2, PHY: 1, PBA No: G73131-008
[ 6.615009] ixgbe 0000:01:00.1: b4:96:91:08:9d:2e
[ 6.624656] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[ 6.625207] libphy: ixgbe-mdio: probed
And finally:
pi@raspberrypi:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b8:27:eb:5c:89:43 brd ff:ff:ff:ff:ff:ff
inet 10.0.100.119/24 brd 10.0.100.255 scope global dynamic noprefixroute eth0
valid_lft 86327sec preferred_lft 75527sec
inet6 fe80::1b5c:e18a:f2b8:7a3b/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b8:27:eb:74:f2:6c brd ff:ff:ff:ff:ff:ff
4: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:96:91:08:9d:2c brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:96:91:08:9d:2e brd ff:ff:ff:ff:ff:ff
I'll be doing some bandwidth tests soon. One thing I might do is see how fast the card can communicate between ports.
I'd assume that #330 and this one are actually the same card. Wanna merge those two? They are confirmed working and I wouldn't see a reason to have the Dell-branded card listed separately
