raspberry-pi-pcie-devices
raspberry-pi-pcie-devices copied to clipboard
Published
20 hours ago •
geerlingguy
geerlingguy
Add Intel 82599ES 10-Gigabit SFI/SFP+ Network Adaptor
trafficstars
tl;dr: Works? Use ixgbe driver, but I wasn't able to do end-to-end test.
Enable the following in menuconfig:
Device Drivers -> Network device support -> Ethernet driver support -> Intel devices -> Intel(R) 10GbE PCI Express adapters support
Reboot with new kernel and should see something like the following via lspci -vk:
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Oracle/SUN Ethernet Server Adapter X520-2
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 90000000 (64-bit, non-prefetchable) [size=512K]
I/O ports at e020 [size=32]
Memory at 90200000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at 90080000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number [REDACTED]
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: ixgbe
Kernel modules: ixgbe
02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Oracle/SUN Ethernet Server Adapter X520-2
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at 90100000 (64-bit, non-prefetchable) [size=512K]
I/O ports at e000 [size=32]
Memory at 90404000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at 90180000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number [REDACTED]
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: ixgbe
Kernel modules: ixgbe
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at df100000 (64-bit, non-prefetchable) [size=1M]
I/O ports at e020 [disabled] [size=32]
Memory at df204000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number [REDACTED]
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: ixgbe
Kernel modules: ixgbe
02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at df000000 (64-bit, non-prefetchable) [size=1M]
I/O ports at e000 [disabled] [size=32]
Memory at df200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number [REDACTED]
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: ixgbe
Kernel modules: ixgbe
Couple of differences to note:
Subsystem: the second output set is from an IBM branded card, but it's the same controller.
The Oracle/SUN card has an extra Expansion ROM entry, and an extra [e0] Vital Product Data capability.
Example output from ip a:
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 90:e2:ba:0a:f5:40 brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 90:e2:ba:0a:f5:41 brd ff:ff:ff:ff:ff:ff
Relevant output from dmesg:
[ 1.390897] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 1.390910] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 1.391604] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 1.391639] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 2.581826] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 2.582151] ixgbe 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 2.582479] ixgbe 0000:01:00.0: MAC: 2, PHY: 1, PBA No: G43015-001
[ 2.582493] ixgbe 0000:01:00.0: [MAC REDACTED]
[ 2.588381] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[ 2.588855] libphy: ixgbe-mdio: probed
[ 2.589329] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[ 3.777672] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 3.777996] ixgbe 0000:01:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 3.778322] ixgbe 0000:01:00.1: MAC: 2, PHY: 1, PBA No: G43015-001
[ 3.778338] ixgbe 0000:01:00.1: [MAC REDACTED]
[ 3.784107] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[ 3.784540] libphy: ixgbe-mdio: probed
[ 3.784646] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver
[ 3.784657] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation.
[ 3.785028] ixgb: Intel(R) PRO/10GbE Network Driver
[ 3.785040] ixgb: Copyright (c) 1999-2008 Intel Corporation.
[ 170.022872] ixgbe 0000:01:00.0 eth1: detected SFP+: 7
Tried to set up a link between the IBM card and a Netgear switch with a Brocade SFP active direct attach cable (see last dmesg entry), but wasn't able to figure out the configuration. May try some actual fiber optics, rule out a bad cable.
The lighting where I was working was terrible, I don't have usable photos at the moment. :c
Did you add the option "allow_unsupported_sfp" for your DAC if not Intel branded? Could be uint (=1) or array (=1,1) depending on ixgbe version. Does SR-IOV work?
It's working here for me: RPi CM4 with Intel 82599ES (using an PCIe x1 to PCIe x8 "adapter" cable -> Cisco DAC -> Cisco 2960X -> Cisco DAC -> PC with 82599ES
dmesg on Pi:
root@raspberrypi:~# dmesg | grep -i ixgbe
[ 8.499279] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 8.499311] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 8.596300] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 9.069951] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 9.070303] ixgbe 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 9.070407] ixgbe 0000:01:00.0: MAC: 2, PHY: 14, SFP+: 3, PBA No: E68793-006
[ 9.070424] ixgbe 0000:01:00.0: 90:e2:ba:89:d3:d8
[ 9.286609] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[ 9.300647] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[ 10.500043] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 10.500388] ixgbe 0000:01:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 10.500492] ixgbe 0000:01:00.1: MAC: 2, PHY: 1, PBA No: E68793-006
[ 10.500508] ixgbe 0000:01:00.1: 90:e2:ba:89:d3:d9
[ 10.533952] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[ 11.407013] ixgbe 0000:01:00.0: registered PHC device on eth1
[ 11.586831] ixgbe 0000:01:00.0 eth1: detected SFP+: 3
[ 11.614290] ixgbe 0000:01:00.1: registered PHC device on eth2
[ 11.656554] ixgbe 0000:01:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 11.657875] ixgbe 0000:01:00.0 eth1: NIC Link is Down
[ 11.939061] ixgbe 0000:01:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Note the line 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
iperf3 results:
root@raspberrypi:~# iperf3 -c 192.168.188.205 -t 5
Connecting to host 192.168.188.205, port 5201
[ 5] local 192.168.188.233 port 58858 connected to 192.168.188.205 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 357 MBytes 2.99 Gbits/sec 25 503 KBytes
[ 5] 1.00-2.00 sec 352 MBytes 2.96 Gbits/sec 36 611 KBytes
[ 5] 2.00-3.00 sec 360 MBytes 3.02 Gbits/sec 0 689 KBytes
[ 5] 3.00-4.00 sec 352 MBytes 2.96 Gbits/sec 3 721 KBytes
[ 5] 4.00-5.00 sec 356 MBytes 2.99 Gbits/sec 3 731 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 1.74 GBytes 2.98 Gbits/sec 67 sender
[ 5] 0.00-5.00 sec 1.73 GBytes 2.98 Gbits/sec receiver
iperf Done.
root@raspberrypi:~# iperf3 -c 192.168.188.205 -t 5 -R
Connecting to host 192.168.188.205, port 5201
Reverse mode, remote host 192.168.188.205 is sending
[ 5] local 192.168.188.233 port 58886 connected to 192.168.188.205 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 368 MBytes 3.09 Gbits/sec
[ 5] 1.00-2.00 sec 368 MBytes 3.09 Gbits/sec
[ 5] 2.00-3.00 sec 368 MBytes 3.09 Gbits/sec
[ 5] 3.00-4.00 sec 368 MBytes 3.09 Gbits/sec
[ 5] 4.00-5.00 sec 368 MBytes 3.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 1.80 GBytes 3.09 Gbits/sec 153 sender
[ 5] 0.00-5.00 sec 1.80 GBytes 3.09 Gbits/sec receiver
iperf Done.
root@raspberrypi:~# iperf3 -c 192.168.188.205 -t 5 --bidir
Connecting to host 192.168.188.205, port 5201
[ 5] local 192.168.188.233 port 60380 connected to 192.168.188.205 port 5201
[ 7] local 192.168.188.233 port 60386 connected to 192.168.188.205 port 5201
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-1.00 sec 282 MBytes 2.37 Gbits/sec 7 607 KBytes
[ 7][RX-C] 0.00-1.00 sec 288 MBytes 2.41 Gbits/sec
[ 5][TX-C] 1.00-2.00 sec 338 MBytes 2.83 Gbits/sec 8 863 KBytes
[ 7][RX-C] 1.00-2.00 sec 151 MBytes 1.27 Gbits/sec
[ 5][TX-C] 2.00-3.00 sec 345 MBytes 2.89 Gbits/sec 8 795 KBytes
[ 7][RX-C] 2.00-3.00 sec 113 MBytes 948 Mbits/sec
[ 5][TX-C] 3.00-4.00 sec 344 MBytes 2.88 Gbits/sec 5 918 KBytes
[ 7][RX-C] 3.00-4.00 sec 109 MBytes 915 Mbits/sec
[ 5][TX-C] 4.00-5.00 sec 346 MBytes 2.90 Gbits/sec 5 962 KBytes
[ 7][RX-C] 4.00-5.00 sec 121 MBytes 1.01 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-5.00 sec 1.62 GBytes 2.78 Gbits/sec 33 sender
[ 5][TX-C] 0.00-5.00 sec 1.61 GBytes 2.77 Gbits/sec receiver
[ 7][RX-C] 0.00-5.00 sec 787 MBytes 1.32 Gbits/sec 46 sender
[ 7][RX-C] 0.00-5.00 sec 782 MBytes 1.31 Gbits/sec receiver
iperf Done.
