raspberry-pi-pcie-devices
raspberry-pi-pcie-devices copied to clipboard
geerlingguy
Test ASUS PCE-AC51 802.11ac WiFi adapter
I just got the ASUS PCE-AC51 802.11ac WiFi adapter, and I'd like to see if it can give me way faster WiFi than the built-in WiFi on the Pi CM4 (which got up to 70-80 Mbps in my testing on my 5 GHz network).
Good news:
$ sudo lspci -vvvv
...
01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8812AE 802.11ac PCIe Wireless Network Adapter (rev 01)
Subsystem: ASUSTeK Computer Inc. RTL8812AE 802.11ac PCIe Wireless Network Adapter
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 255
Region 0: I/O ports at <unassigned> [disabled]
Region 2: Memory at 600000000 (64-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Via message/WAKE#
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 00-12-88-fe-ff-4c-e0-00
Capabilities: [150 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [158 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
And:
[ 1.010408] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.010445] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.010520] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[ 1.010592] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x00ffffffff -> 0x0100000000
[ 1.043164] brcm-pcie fd500000.pcie: link up, 2.5 GT/s x1 (SSC)
[ 1.043469] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 1.043499] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.043527] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[ 1.043600] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 1.043834] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 1.047448] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[ 1.047661] pci 0000:01:00.0: [10ec:8812] type 00 class 0x028000
[ 1.047752] pci 0000:01:00.0: reg 0x10: [io 0x0000-0x00ff]
[ 1.047823] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00003fff 64bit]
[ 1.048078] pci 0000:01:00.0: supports D1 D2
[ 1.048101] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.051575] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 1.051628] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[ 1.051662] pci 0000:01:00.0: BAR 2: assigned [mem 0x600000000-0x600003fff 64bit]
[ 1.051715] pci 0000:01:00.0: BAR 0: no space for [io size 0x0100]
[ 1.051741] pci 0000:01:00.0: BAR 0: failed to assign [io size 0x0100]
[ 1.051767] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 1.051798] pci 0000:00:00.0: bridge window [mem 0x600000000-0x6000fffff]
Also, this card did not work through my IO Crest PCIe bridge. Though my Syba USB 3.0 card and NVMe drive did. So something to think about: PCIe is not a magic bullet on the Pi 4, and a lot of assumptions people may have coming from the X86 or $500+ ARM board world are not going to hold up.
Trying driver in menuconfig in Device Drivers > Network device support > Wireless LAN > Realtek 802.11ac wireless chips support — compiling the kernel now...
Hmm... looks like the driver rtl8xxxu doesn't do anything with the card. Does it only work over USB?
Maybe I need the rtw88 driver instead? https://github.com/torvalds/linux/commit/e3037485c68ec1a299ff41160d8fedbd4abc29b9
Going to give https://github.com/lwfinger/rtw88 a spin (found after glancing through https://github.com/gnab/rtl8812au and the rtw88 kernel driver):
sudo apt-get install -y make gcc raspberrypi-kernel-headers build-essential git
git clone https://github.com/lwfinger/rtw88.git
cd rtw88
make
sudo make install
On the make command it stops, likely due to https://github.com/raspberrypi/Raspberry-Pi-OS-64bit/issues/4
$ make
make -C /lib/modules/5.4.51-v8+/build M=/home/pi/rtw88 modules
make[1]: *** /lib/modules/5.4.51-v8+/build: No such file or directory. Stop.
make: *** [Makefile:79: all] Error 2
I'm going to quickly try installing Ubuntu 20.04 (Server, don't need the gooky GUI bits) to see if I can get a quicker start there. I believe the kernel headers are more standard on that distro, but I could just be recklessly optimistic there :P
EDIT: OOH, I never realized Ubuntu for Pi includes cloud-init (see /Volumes/system-boot/user-data file), so I can add in the following to get my keys from GitHub inserted in the default user account so I can ssh without having to spend a few seconds using ssh-copy-id the first time like with Pi OS...
ssh_import_id:
- gh:geerlingguy
Edit 2: I also always forget that the ubuntu image by default runs an unattended upgrade on first boot... which takes a nice loooong time to run. So waiting for that to finish...
On Ubuntu:
sudo apt-get install -y make gcc linux-headers-$(uname -r) build-essential git
Results in:
update-initramfs: Generating /boot/initrd.img-5.4.0-1015-raspi
Unsupported platform.
run-parts: /etc/initramfs/post-update.d//flash-kernel exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
So... drat. Luckily, it seems to get far enough that I can make the rtw88 project. So I'm trying that now:
git clone https://github.com/lwfinger/rtw88.git
cd rtw88
make
sudo make install
Well, unfortunately that results in:
$ sudo make install
make -C /lib/modules/5.4.0-1015-raspi/build M=/home/ubuntu/rtw88 modules
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-1015-raspi'
Building modules, stage 2.
MODPOST 10 modules
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-1015-raspi'
Making backups
tar: /lib/modules/5.4.0-1015-raspi/kernel/drivers/net/wireless/realtek/rtw88: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
make: *** [Makefile:83: install] Error 2
I might have to throw in the towel on this WiFi adapter. Many people seem to think Intel chips are the way to go for better Linux support, and they may be right.
Switching gears to https://github.com/gnab/rtl8812au —
sudo apt-get install -y make gcc linux-headers-$(uname -r) build-essential git
git clone https://github.com/gnab/rtl8812au
# Edit the Makefile and set the following:
CONFIG_PLATFORM_I386_PC = n
CONFIG_PLATFORM_ARM_RPI = y
make
That failed with:
$ make
make ARCH=arm CROSS_COMPILE= -C /lib/modules/5.4.0-1015-raspi/build M=/home/ubuntu/rtl8812au modules
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-1015-raspi'
CC [M] /home/ubuntu/rtl8812au/core/rtw_cmd.o
gcc: error: unrecognized argument in option ‘-mabi=apcs-gnu’
gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
gcc: error: unrecognized command line option ‘-mapcs’
gcc: error: unrecognized command line option ‘-mno-sched-prolog’
gcc: error: unrecognized command line option ‘-msoft-float’
make[2]: *** [scripts/Makefile.build:275: /home/ubuntu/rtl8812au/core/rtw_cmd.o] Error 1
make[1]: *** [Makefile:1734: /home/ubuntu/rtl8812au] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-1015-raspi'
make: *** [Makefile:1068: modules] Error 2
But I found this issue (https://github.com/gnab/rtl8812au/issues/25#issuecomment-527273047), and that seemed to allow things to proceed:
$ make ARCH="arm64"
...
CC [M] /home/ubuntu/rtl8812au/core/rtw_mp_ioctl.o
LD [M] /home/ubuntu/rtl8812au/8812au.o
Building modules, stage 2.
MODPOST 1 modules
CC [M] /home/ubuntu/rtl8812au/8812au.mod.o
LD [M] /home/ubuntu/rtl8812au/8812au.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-1015-raspi'
I did get one compile error on rtw_android.c about an unsigned int, but it seems like that should be fine.
I ran sudo insmod 8812au.ko and now dmesg shows:
[ 1865.996258] usbcore: registered new interface driver rtl8812au
Oops. Might need to tweak the config to use PCI instead of USB...
Removing module with sudo rmmod 8812au.ko, then editing the Makefile with changes:
CONFIG_USB_HCI = n
CONFIG_PCI_HCI = y
But now when I build I get:
CC [M] /home/ubuntu/rtl8812au/os_dep/linux/os_intfs.o
make[2]: *** No rule to make target '/home/ubuntu/rtl8812au/os_dep/linux/pci_intf.o', needed by '/home/ubuntu/rtl8812au/8812ae.o'. Stop.
make[1]: *** [Makefile:1734: /home/ubuntu/rtl8812au] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-1015-raspi'
make: *** [Makefile:1068: modules] Error 2
Maybe I'm hitting this: https://unix.stackexchange.com/questions/591574/installing-realtek-8821ce-on-rhel-8#comment1103779_591677
It seems like this driver might just not support PCIe at all?
All right, now trying https://github.com/gordboy/rtl8812au-5.6.4.2 — same dealio... edit Makefile, disable USB, enable PCI, disable x386, enable RPI, save it then compile with make ARCH="arm64"... waiting, and...
CC [M] /home/ubuntu/rtl8812au-5.6.4.2/os_dep/linux/os_intfs.o
make[2]: *** No rule to make target '/home/ubuntu/rtl8812au-5.6.4.2/os_dep/linux/pci_intf.o', needed by '/home/ubuntu/rtl8812au-5.6.4.2/8812ae.o'. Stop.
make[1]: *** [Makefile:1734: /home/ubuntu/rtl8812au-5.6.4.2] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-1015-raspi'
make: *** [Makefile:2271: modules] Error 2
Same error. Methinks nobody uses PCIe WiFi cards???
Tried with https://github.com/aircrack-ng/rtl8812au (editing Makefile accordingly) ... and same issue:
$ make ARCH="arm64"
...
make[2]: *** No rule to make target '/home/ubuntu/rtl8812au/os_dep/linux/_intf.o', needed by '/home/ubuntu/rtl8812au/8812ae.o'. Stop.
make[1]: *** [Makefile:1734: /home/ubuntu/rtl8812au] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-1015-raspi'
make: *** [Makefile:2247: modules] Error 2
Note that it seems this fork of the driver is messed with a bit extra, so it had even more of the PCI bits removed even from the Makefile.
Might try using the iwlwifi driver and something like this EDUP PCIe Intel AX200 WiFi 6 Card instead...
Interesting..... it appears everyone forked the base realtek driver and worked from there, so the makefile still holds a PCI profile, but nobody bothered to implement intf_pci.c in their forks. It does appear to exist in the driver source for the older realteks
update-initramfs: Generating /boot/initrd.img-5.4.0-1015-raspi Unsupported platform. run-parts: /etc/initramfs/post-update.d//flash-kernel exited with return code 1
You tried using strace to track where its failing? A quick google indicates it could be anything from the device tree not being properly populated, recent updates to how model numbers are represented in RPi firmware, to the platform itself not requiring flash-kernel
EDIT: Apparently Cannonical themselves are aware of the issue and have a "Confirmed" bug report on the matter that has been open since 2018. Not sure if their official Ubuntu guide is updated, but they indicate that for now, just update the flash-kernel database yourself on your device
@PixlRainbow Hmm, good find! I might have to dig back in with this card later—right now I'm going to put it on pause and work on testing a card I just got in the mail with the AX200 Intel chipset :)
Those rtl8812au drivers are pretty tricky to get it to work. You need to find the right version of it and tune the Makefile to target ARM.
Video posted: WiFi 6 on the Raspberry Pi CM4 makes it Fly! MORE THAN 1 Gbps! (Note that the video doesn't dig into this card too deeply, but I may take another crack at it.)
At about 15:50 into the video you show the 10g mikrotik crs305 and say that it can't do 2.5G/5G, but does just 1G or 10G. That's not quite true, as it can negotiate 2.5G or 5G data rates, as long as your SFP+ module can.
Serve The Home tested a bunch of SFP+ 10Gbase-T modules and shows that not only do some not support the 2.5/5 standards, but those that do are faster at 10G overall: https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/
In short, ensure you get a 10GBase-T module that can do 2.5GBase-T and 5GBase-T and it should work at the right speed in the Mikrotik.
@AshleyPinner - Thanks! I actually found that a different transceiver supports all the speeds. An older one I was testing only did 1/10 Gbps.
If someone else wants to put in some work seeing if they can get this working, I'm open to it, but otherwise I'm going to chalk this card up as 'not going to spend more time trying to get it working because I'd rather recommend AX200-based cards instead :)
If you do a "lspci -vn" to get the PCI IDs, I can tell you which driver you need for Linux.
@jcdutton - See below:
$ sudo lspci -vn
00:00.0 0604: 14e4:2711 (rev 20) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 62
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: c0000000-c00fffff
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
Capabilities: [240] L1 PM Substates
Kernel driver in use: pcieport
01:00.0 0280: 10ec:8812 (rev 01)
Subsystem: 1043:86dd
Flags: fast devsel, IRQ 255
I/O ports at <unassigned> [disabled]
Memory at 600000000 (64-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-12-88-fe-ff-4c-e0-00
Capabilities: [150] Latency Tolerance Reporting
Capabilities: [158] L1 PM Substates
Use: modprobe rtl8821ae
The 21 is not a typo. The driver supports the 8821 and the 8812 chips.
@jcdutton - With the base Pi OS install I get:
$ modprobe rtl8821ae
modprobe: FATAL: Module rtl8821ae not found in directory /lib/modules/5.10.35-v7l+
So I'm going to recompile the kernel with:
Device Drivers
> Network device support
> Wireless LAN
> Realtek rtlwifi family of devices
> Realtek RTL8821AE/RTL8812AE Wireless Network Adapter
(As seen in https://cateee.net/lkddb/web-lkddb/RTL8821AE.html).
We'll see how it goes...
make menuconfig Device Drivers ---> -- Network device support ---> [] Wireless LAN ---> <M> Realtek rtlwifi family of devices ---> <M> Realtek RTL8821AE/RTL8812AE Wireless Network Adapter
01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8812AE 802.11ac PCIe Wireless Network Adapter (rev 01)
Subsystem: ASUSTeK Computer Inc. RTL8812AE 802.11ac PCIe Wireless Network Adapter
Flags: fast devsel, IRQ 47
I/O ports at 0000
Memory at 600000000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-12-88-fe-ff-4c-e0-00
Capabilities: [150] Latency Tolerance Reporting
Capabilities: [158] L1 PM Substates
Kernel modules: rtl8821ae
And:
$ lsmod | grep rtl8821ae
rtl8821ae 253952 0
btcoexist 196608 1 rtl8821ae
rtl_pci 36864 1 rtl8821ae
rtlwifi 118784 3 rtl_pci,rtl8821ae,btcoexist
mac80211 905216 3 rtl_pci,rtl8821ae,rtlwifi
But if I list interfaces it's not in the mix:
$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b8:27:eb:5c:89:43 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DORMANT group default qlen 1000
link/ether b8:27:eb:74:f2:6c brd ff:ff:ff:ff:ff:ff
(The wlan0 is the built-in wlan).
$ iw dev
phy#1
Unnamed/non-netdev interface
wdev 0x100000002
addr ba:27:eb:74:f2:6c
type P2P-device
Interface wlan0
ifindex 3
wdev 0x100000001
addr b8:27:eb:74:f2:6c
type managed
channel 34 (5170 MHz), width: 20 MHz, center1: 5170 MHz
Does dmesg give any output relating to it? At least the kernel agrees that it is the right driver. It might be looking for: /lib/firmware/rtlwifi/rtl8812aefw.bin /lib/firmware/rtlwifi/rtl8812aefw_wowlan.bin
@jcdutton Ah... silly me, didn't even think to look there:
$ dmesg | grep rtl
[ 5.122777] rtl8821ae 0000:01:00.0: enabling device (0000 -> 0002)
[ 5.181119] rtl8821ae: Using firmware rtlwifi/rtl8812aefw.bin
[ 5.182443] rtl8821ae 0000:01:00.0: Direct firmware load for rtlwifi/rtl8812aefw.bin failed with error -2
[ 5.183980] rtlwifi: Loading alternative firmware rtlwifi/rtl8821aefw.bin
[ 5.184083] rtl8821ae: Using firmware rtlwifi/rtl8812aefw_wowlan.bin
[ 5.184215] rtl8821ae 0000:01:00.0: Direct firmware load for rtlwifi/rtl8812aefw_wowlan.bin failed with error -2
[ 5.184337] rtlwifi: Loading alternative firmware rtlwifi/rtl8821aefw.bin
[ 5.189198] rtl_pci: Cannot allocate RX ring
[ 5.189218] rtl_pci: tx ring initialization failed
[ 5.189228] rtl_pci: Failed to init PCI
[ 5.189357] rtl8821ae: probe of 0000:01:00.0 failed with error -12
Those firmware files are in the linux-firmware .deb package. Just copy them into the /lib/firmware/rtfwifi folder. They should end up here: /lib/firmware/rtlwifi/rtl8812aefw.bin /lib/firmware/rtlwifi/rtl8812aefw_wowlan.bin
These firmware files are the same for both x86 and arm. Once the firmware files are in place. Reboot and the system should just see the new wlan1 interface.
I'm pretty sure the firmware was loaded fine though: rtlwifi: Loading alternative firmware rtlwifi/rtl8821aefw.bin
And I checked and firmware-realtek is already installed...