pwnagotchi
pwnagotchi copied to clipboard
evilsocket
nexmon blindness bug (brcmf_cfg80211_nexmon_set_channel)
every once in a while, nexmon dies with:
[ 4341.527847] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 4344.327806] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 4347.127853] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 4349.927917] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 4352.728074] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 4355.527970] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 4358.328022] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 4361.208095] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 4364.008157] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 4366.808218] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 4369.608431] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 4372.408345] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 4375.288408] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 4378.088474] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 4380.891399] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
And only a reboot can fix the wifi, this is why the mon_max_blind_epochs parameter exists, to reboot the board when this happens.
Ideally we should document this known issue, the configuration and some day maybe fix it.
You could try reloading the driver instead of rebooting the pi, using :
modprobe -r brcmfmac
modprobe brcmfmac
If this still fails, then a reboot will fix this. There is a great discussion about that problem here : https://www.bountysource.com/issues/56252669-wlan-freezes-in-raspberry-pi-3b
tried that way, it doesn't always work, the only reliable way is rebooting
So, The Nexmon firmware is a little picky on how its interfaced, How are your bringing up the mon interface in linux before bettercap gets to it?
from https://github.com/evilsocket/pwnagotchi/blob/master/builder/pwnagotchi.yml#L333
I just noticed on the Nexmon Repo that the bcm43455c0 does not support wifi frame injection. That might be the reason for the drivers crashing as they are not handling the requests to do frame injection correctly from bettercap.
Ok, So I looked, We are using the older version of the firmware provided by Nexmon, I think Re4son kernel does it as its the default for Nexmon to use that. Patches and Commits from the Nexmon Project show 7.45.189 as the latest version you can use (The base firmware comes from the OEM, not the RPI foundation) as of RC4, we are using version 7.45.154 of the bcm43455c0 firmware This issue should only affect 3B+ and 4s, the 3B and the 0W use the same wifi chip and are listed as supporting injection.
More Details, I've looked into how the Re4son kernel builder pulls down its firmware, its even /worse/ Looks like it pulls from https://github.com/Re4son/re4son-nexmon as its source of nexmon patches... its years out of date. I'm doing some prototyping to update this now.
Good News, The Nexmon Patches with the stock kernel works well, Here is my DMesg output after running for 10 minutes, I will be running it for the next 24 hours to see if its stable,
[ 5.128852] brcmfmac: loading out-of-tree module taints kernel.
[ 5.128864] brcmfmac: loading out-of-tree module taints kernel.
[ 5.181306] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 5.190412] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 5.190907] usbcore: registered new interface driver brcmfmac
[ 5.566003] brcmfmac: brcmf_sdio_bus_preinit: before brcmf_sdio_debugfs_create
[ 5.569776] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 5.597271] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Oct 15 2019 20:30:25 version 7.45.189 (nexmon.org: -4) FWID 01-e1db26e2
[ 5.675296] brcmfmac: brcmf_bus_started: before brcmf_debugfs_add_entry
[ 8.495009] brcmfmac: brcmf_vif_add_validate: Attempt to add a MONITOR interface...
[ 8.495024] brcmfmac: brcmf_mon_add_vif: brcmf_mon_add_vif called
[ 8.495028] brcmfmac: brcmf_mon_add_vif: Adding vif "mon0"
Here is my uname
Linux pwnagotchi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux
And I failed, You get a little more debug output this time overall..
[ 5387.511962] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 5390.551934] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 5393.601927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5396.631949] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 5399.671984] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5402.711975] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5405.752020] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 5408.791991] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5411.831985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4106, -110
[ 5414.872013] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53284, -110
[ 5417.911966] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53288, -110
[ 5420.951985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53296, -110
[ 5423.511967] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5426.312006] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 5429.111938] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5431.911970] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5434.711974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53308, -110
[ 5437.511963] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5440.311927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53356, -110
[ 5443.111922] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53360, -110
[ 5445.911981] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53368, -110
[ 5448.711973] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53372, -110
[ 5451.511925] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53380, -110
[ 5454.311928] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5457.111929] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 5459.911983] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5462.711974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5465.511924] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53308, -110
[ 5468.311926] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5471.111925] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53356, -110
[ 5473.912031] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53360, -110
[ 5476.471932] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5479.271932] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5482.081987] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 5484.871994] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5487.671927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53292, -110
[ 5490.471930] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53300, -110
[ 5493.271977] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53304, -110
[ 5496.071923] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5498.872004] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53364, -110
[ 5501.671976] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53372, -110
[ 5504.471928] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53376, -110
[ 5507.031989] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 5507.032002] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
[ 5509.591930] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 5509.591939] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
[ 5512.151989] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5514.952027] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
Edit: Added brcmfmac.debug=30 to /boot/cmdline.txt to try and get a better look at what the driver is doing, It enables full trace mode for the drivers, I have attacked a full debug log up to where it strops responding. syslog.gz
yep when that happens even trying to change channel manually doesn't work, i think it's the heat
Based off this photo, There is no TIM under the wifi can at all... Since its used for shielding RF, this is of no surprise. Someone should get a IR Camera on the board and take a look.
Also You notice that White IC in the top left, Thats the same chip they used to show people the SuperMicro Implant... lulz
So, I'm looking at the datasheet, for the CYW43455 It states that the Max temp for operation is 120C while under normal loads on a 4 layer board Of course Max oper temp is +85C The thing only puts out 1.2W but I did notice that on the older RPI0W Chips it had a self limiter for overheat, This one is not stated in the data sheet as having one
so you're saying that it's the chinese sabotaging our wifi pwning, right?
Maybe.... Until I can get a proper Temp readout of the die itself while under the heavy load, We wont know.
Hi, is the same issue afflicting both 43455c0 and 43430a1? Have you got a reliable way to reproduce it?
@evilsocket hey, I am investigating "brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110" aka Nexmon blindness bug issue and have noticed something that can potentially lead to resolution of the problem. My understanding was that patched Nexmon failed to set a channel, so instead of hopping on all channels I specified the channels manually to use for sending deauth frames in config.yml. In my case most AP are sitting on 1,4,6 and 11. Gotchi has been running for 15 minutes so far without any crashes. Can somebody confirm if it can be a valid solution ?
@hackabean well not really, you just worked around the issue, but when that happens the whole wifi chip stops being very responsive
if you specify the channels manually it will still hop ... i wonder if that's an issue related to hopping too quickly ...
Took the gotchi for a walk in the busy neighborhood, have set the channels beforehand and the issue still remains. Got excited prematurely, this does not work either, sorry :(
@hackabean -110 means firmware hanged. When working on nexmon I experienced blindness followed by cmds timeout when I was leaking memory in rx queue. Timeout/traps are fw bugs.
@DrSchottky thanks for stepping in, seems like you know more about this issue. I just wished I could do something to make this less problematic so I was just poking around hoping to find a workaround. I am crossing my fingers that somebody will take a look at it.
@hackabean to work on the problem I need to know as much as possible about the setup (board and kernel/driver/fw versions), the environment (does it crash under particular conditions? How often?) and I need to have a reliable way to reproduce the bug on my testbed. AFAIK monitor mode should be stable (never used injection in deep so far)
@DrSchottky I can reproduce it pretty reliably on my Pi4 setup. I may need a little help gathering the info you need, but I'm pretty confident it will go blind within an hour every time I power it up. What can I do to help?
