firmware
firmware copied to clipboard
raspberrypi
Wifi: Firmware crashing regularly on RPi 0W2
Describe the bug The firmware of the onboard WiFi on my RPi 0W2 crashes almost every day:
[78167.189463] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
[78167.237807] brcmfmac: CONSOLE:
[78167.237835] brcmfmac: CONSOLE: FWID 01-3b307371
[78167.237846] brcmfmac: CONSOLE: flags 1
[78167.237856] brcmfmac: CONSOLE:
[78167.237868] brcmfmac: CONSOLE: TRAP c(6ff7c): pc 837364, lr 8372bb, sp 3f828, cpsr 4000000c, spsr 41000000
[78167.237881] brcmfmac: CONSOLE: r0 0, r1 4, r2 0, r3 0, r4 0, r5 0, r6 60cbc
[78167.237893] brcmfmac: CONSOLE: r7 4f390, r8 61190, r9 0, r10 168, r11 0, r12 7c
[78167.237904] brcmfmac: CONSOLE:
[78167.237916] brcmfmac: CONSOLE: sp+0 00000000 00802fa9 00060de8 00061218
[78167.237928] brcmfmac: CONSOLE: sp+10 00065e00 00061190 00065e00 00060cbc
[78167.237939] brcmfmac: CONSOLE:
[78167.237949] brcmfmac: CONSOLE: sp+4 00802fa9
[78167.237959] brcmfmac: CONSOLE: sp+34 00019ab5
[78167.237970] brcmfmac: CONSOLE: sp+5c 00844f3f
[78167.237980] brcmfmac: CONSOLE: sp+7c 008472b5
[78167.237991] brcmfmac: CONSOLE: sp+94 0085e3f5
[78167.238001] brcmfmac: CONSOLE: sp+fc 0081e9e9
[78167.238012] brcmfmac: CONSOLE: sp+130 00844c95
[78167.238023] brcmfmac: CONSOLE: sp+134 00844fe9
[78167.238033] brcmfmac: CONSOLE: sp+164 0002f6df
[78167.238044] brcmfmac: CONSOLE: sp+1bc 00807f53
[78167.238055] brcmfmac: CONSOLE: sp+1f4 00875b39
[78167.238065] brcmfmac: CONSOLE: sp+234 0001b6b1
[78167.238076] brcmfmac: CONSOLE: sp+274 0001e085
[78167.238087] brcmfmac: CONSOLE: sp+29c 0082eed3
[78167.238097] brcmfmac: CONSOLE: sp+2c4 0082f215
[78167.238108] brcmfmac: CONSOLE: sp+2d4 0081f351
[78167.285400] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[78167.285433] ieee80211 phy0: brcmf_cfg80211_get_tx_power: error (-5)
[78167.937870] brcmfmac: brcmf_sdiod_probe: Failed to set F1 blocksize
[78167.937934] brcmfmac: brcmf_sdio_bus_reset: Failed to probe after sdio device reset: ret -123
It seems I can only get it back online by restarting the RPi. I am tracking this issue for the past months and already tried running with over_voltage=2 as suggested elsewhere, but no luck.
Expected behaviour WiFi firmware does not crash.
Actual behaviour WiFi firmware crashes.
System & Logs See the attached raspinfo.txt.
Additional context I can't confirm it 100% yet, but the crash seems to happen whenever the connection is handed over from my WiFi router to my WiFi repeater or vice versa.
There's a ticket open with Synaptics for this issue. I'll report back with any questions or findings.
I had the same issue with a Raspberry Zero 2 W and Dietpi OS. With like every day unrecoverable brcmf_fw_crash. It seems its related to overlapping wifi networks. We used 2 routers here with the same SSID. I changed one of them SSID to something else, and since then I didnt see wifi issue. I installed a bash script to check my router ip access every 5 min and restart kernel module if needed but it wasnt. What is weird I didnt upgrade the Raspberry packages (Dietpi OS) and we used this router setup long time ago, so I dont know the reason why it started facing the issue some weeks ago.
Example for the crash: Dec 29 19:00:10 zero2 kernel: [122694.272662] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled Dec 29 19:00:11 zero2 kernel: [122695.912670] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed Dec 29 19:00:12 zero2 kernel: [122695.985029] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do. Dec 29 19:00:12 zero2 kernel: [122695.985074] ieee80211 phy0: brcmf_cfg80211_get_tx_power: error (-5) Dec 29 19:00:12 zero2 kernel: [122696.617005] brcmfmac: brcmf_sdiod_probe: Failed to set F1 blocksize Dec 29 19:00:12 zero2 kernel: [122696.617137] brcmfmac: brcmf_sdio_bus_reset: Failed to probe after sdio device reset: r et -123 Dec 29 19:00:12 zero2 kernel: [122696.618115] mmc1: card 0001 removed Dec 29 19:00:12 zero2 kernel: [122696.660762] mmc1: queuing unknown CIS tuple 0xc0 [dd 38 c1 95 4f f1 8a 61 89 b7 68 bf e2 45 00] (15 bytes) Dec 29 19:00:12 zero2 kernel: [122696.751641] mmc1: queuing unknown CIS tuple 0x2a [80 ff e6 ec 00 00 01 0a 1e 2c 00 f9 92 b6 b4 0c ff 01 13 48 ff 00 e4 41 48 09 38 6b 4c 1f ff 00 58 0a 12 83 7c 7f f4 00 ff ad ff ec 22 08 c6 59 df 10 f8 33 da 1f 48 ff a5 e2 28 2f b4 ff f0 20] (253 bytes) Dec 29 19:00:12 zero2 kernel: [122696.753777] mmc1: queuing unknown CIS tuple 0xf1 [03 35] (2 bytes) Dec 29 19:00:12 zero2 kernel: [122696.805589] mmc1: queuing unknown CIS tuple 0x2e [e2 12 22 c1 04 72 76 ff 8d ff fe a4 49 44 00 00 4b 16 86 12 c3 3d 75 07 93 00 01 b1 62 ed 2e 49 16 ff 27 13 6a 0b 02 30 06 b0 c2 02 16 7f 00 41 7a 48 6d 02 24 61 00 20 00 00 9f 42 99 06 c1 04] (144 bytes) Dec 29 19:00:57 zero2 kernel: [122740.941905] mmc1: error -22 whilst initialising SDIO card Dec 29 19:00:57 zero2 kernel: [122740.997200] mmc1: queuing unknown CIS tuple 0xc0 [dd 38 c1 95 4f f1 8a 61 89 b7 68 bf e2 45 00] (15 bytes) Dec 29 19:00:57 zero2 kernel: [122741.110450] mmc1: queuing unknown CIS tuple 0x2a [80 ff e6 ec 00 00 01 0a 1e 2c 00 f9 92 b6 b4 0c ff 01 13 48 ff 00 e4 41 48 09 38 6b 4c 1f ff 00 58 0a 12 83 7c 7f f4 00 ff ad ff ec 22 08 c6 59 df 10 f8 33 da 1f 48 ff a5 e2 28 2f b4 ff f0 20] (253 bytes) Dec 29 19:00:57 zero2 kernel: [122741.112705] mmc1: queuing unknown CIS tuple 0xf1 [03 35] (2 bytes) Dec 29 19:00:57 zero2 kernel: [122741.177792] mmc1: queuing unknown CIS tuple 0x2e [e2 12 22 c1 04 72 76 ff 8d ff fe a4 49 44 00 00 4b 16 86 12 c3 3d 75 07 93 00 01 b1 62 ed 2e 49 16 ff 27 13 6a 0b 02 30 06 b0 c2 02 16 7f 00 41 7a 48 6d 02 24 61 00 20 00 00 9f 42 99 06 c1 04] (144 bytes) Dec 29 19:01:52 zero2 kernel: [122796.785417] mmc1: error -22 whilst initialising SDIO card Dec 29 19:01:52 zero2 kernel: [122796.850698] mmc1: queuing unknown CIS tuple 0xc0 [dd 38 c1 95 4f f1 8a 61 89 b7 68 bf e2 45 00] (15 bytes)
Can you do what @porst17 did and add brcmfmac.debug=0x100000 to /boot/cmdline.txt (keeping it all in one line) and reboot? Any firmware crash should then generate a log like the one in the first post:
[78167.189463] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
[78167.237807] brcmfmac: CONSOLE:
[78167.237835] brcmfmac: CONSOLE: FWID 01-3b307371
[78167.237846] brcmfmac: CONSOLE: flags 1
[78167.237856] brcmfmac: CONSOLE:
[78167.237868] brcmfmac: CONSOLE: TRAP c(6ff7c): pc 837364, lr 8372bb, sp 3f828, cpsr 4000000c, spsr 41000000
[78167.237881] brcmfmac: CONSOLE: r0 0, r1 4, r2 0, r3 0, r4 0, r5 0, r6 60cbc
[78167.237893] brcmfmac: CONSOLE: r7 4f390, r8 61190, r9 0, r10 168, r11 0, r12 7c
...
I also want to add that my setup uses the same SSID for the main router and the repeater. This is the default setup of the wifi mesh system (AVM Fritz!Box) I have been using for years without problems on the many WiFi devices I own. It's just the RPi 0W2 that doesn't seem to like it. 🤷♂️
I have also been experiencing this on my 0W2 for two weeks, ever since I moved the pi from a wall socket to a power strip, so it might have something to do with voltage. I use the "official" adapter. Until a couple of days ago I handled it by plugging the pi out and then back in. The firmware could hold up for at least a day before crashing again. Then I switched to a cron script that reboots whenever the firmware crashes, and in the past 24 hours it has crashed 3 times. One thing I notice is that the crash always happens after attempting to renew the DHCP lease. Meanwhile, as other folks, I'm also on a mesh Wi-Fi network (Deco). I'll set the firmware to debug mode and post logs here the next time it crashes.
Logs from a recent crash:
Jan 18 17:50:25 homebridge kernel: [87244.740253] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
Jan 18 17:50:25 homebridge kernel: [87244.798582] brcmfmac: CONSOLE:
Jan 18 17:50:25 homebridge kernel: [87244.798611] brcmfmac: CONSOLE: FWID 01-3b307371
Jan 18 17:50:25 homebridge kernel: [87244.798624] brcmfmac: CONSOLE: flags 1
Jan 18 17:50:25 homebridge kernel: [87244.798636] brcmfmac: CONSOLE:
Jan 18 17:50:25 homebridge kernel: [87244.798648] brcmfmac: CONSOLE: TRAP 3(6ff7c): pc 87b7c6, lr 867231, sp 3f71c, cpsr 20000003, spsr 21000000
Jan 18 17:50:25 homebridge kernel: [87244.798661] brcmfmac: CONSOLE: r0 f7d19ee4, r1 61190, r2 3f808, r3 0, r4 3f784, r5 1002, r6 3f780
Jan 18 17:50:25 homebridge kernel: [87244.798675] brcmfmac: CONSOLE: r7 0, r8 3f808, r9 0, r10 61190, r11 6058c, r12 3f7d4
Jan 18 17:50:25 homebridge kernel: [87244.798687] brcmfmac: CONSOLE:
Jan 18 17:50:25 homebridge kernel: [87244.798699] brcmfmac: CONSOLE: sp+0 0003f784 00001002 00867231 00000001
Jan 18 17:50:25 homebridge kernel: [87244.798711] brcmfmac: CONSOLE: sp+10 0085ec05 ffffffff 0005a080 00000000
Jan 18 17:50:25 homebridge kernel: [87244.798723] brcmfmac: CONSOLE:
Jan 18 17:50:25 homebridge kernel: [87244.798734] brcmfmac: CONSOLE: sp+8 00867231
Jan 18 17:50:25 homebridge kernel: [87244.798746] brcmfmac: CONSOLE: sp+10 0085ec05
Jan 18 17:50:25 homebridge kernel: [87244.798758] brcmfmac: CONSOLE: sp+48 00867607
Jan 18 17:50:25 homebridge kernel: [87244.798769] brcmfmac: CONSOLE: sp+88 008364af
Jan 18 17:50:25 homebridge kernel: [87244.798780] brcmfmac: CONSOLE: sp+a0 00839441
Jan 18 17:50:25 homebridge kernel: [87244.798792] brcmfmac: CONSOLE: sp+b8 008566f3
Jan 18 17:50:25 homebridge kernel: [87244.798803] brcmfmac: CONSOLE: sp+d8 0084807f
Jan 18 17:50:25 homebridge kernel: [87244.798815] brcmfmac: CONSOLE: sp+110 00830fc9
Jan 18 17:50:25 homebridge kernel: [87244.798826] brcmfmac: CONSOLE: sp+15c 00001531
Jan 18 17:50:25 homebridge kernel: [87244.798838] brcmfmac: CONSOLE: sp+164 00011531
Jan 18 17:50:25 homebridge kernel: [87244.798850] brcmfmac: CONSOLE: sp+1c4 0087702d
Jan 18 17:50:25 homebridge kernel: [87244.798861] brcmfmac: CONSOLE: sp+1c8 0087710d
Jan 18 17:50:25 homebridge kernel: [87244.798872] brcmfmac: CONSOLE: sp+1d0 00807b3f
Jan 18 17:50:25 homebridge kernel: [87244.798884] brcmfmac: CONSOLE: sp+230 0000586f
Jan 18 17:50:25 homebridge kernel: [87244.798895] brcmfmac: CONSOLE: sp+28c 008302d5
Jan 18 17:50:25 homebridge kernel: [87244.798907] brcmfmac: CONSOLE: sp+2a0 00833417
Jan 18 17:50:25 homebridge kernel: [87245.147269] ieee80211 phy0: brcmf_netdev_start_xmit: xmit rejected state=0
Jan 18 17:50:27 homebridge kernel: [87247.291281] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Jan 18 17:50:27 homebridge kernel: [87247.292223] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Jan 18 17:50:27 homebridge kernel: [87247.292588] brcmfmac: dongle trap info: type 0x3 @ epc 0x0087b7c6
Jan 18 17:50:27 homebridge kernel: [87247.292588] cpsr 0x20000003 spsr 0x21000000 sp 0x0003f71c
Jan 18 17:50:27 homebridge kernel: [87247.292588] lr 0x00867231 pc 0x0087b7c6 offset 0x6ff7c
Jan 18 17:50:27 homebridge kernel: [87247.292588] r0 0xf7d19ee4 r1 0x00061190 r2 0x0003f808 r3 0x00000000
Jan 18 17:50:27 homebridge kernel: [87247.292588] r4 0x0003f784 r5 0x00001002 r6 0x0003f780 r7 0x00000000
Jan 18 17:50:27 homebridge kernel: [87247.292660] ieee80211 phy0: brcmf_cfg80211_disconnect: error (-110)
Jan 18 17:50:27 homebridge kernel: [87247.292753] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Jan 18 17:50:27 homebridge kernel: [87247.292777] ieee80211 phy0: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, err=-5
Jan 18 17:50:27 homebridge kernel: [87247.292805] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Jan 18 17:50:27 homebridge kernel: [87247.292827] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Jan 18 17:50:28 homebridge kernel: [87247.807106] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Jan 18 17:50:28 homebridge kernel: [87247.807172] ieee80211 phy0: brcmf_cfg80211_get_tx_power: error (-5)
Jan 18 17:50:28 homebridge kernel: [87247.807612] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
Jan 18 17:50:28 homebridge kernel: [87247.807643] ieee80211 phy0: brcmf_cfg80211_reg_notifier: Country code iovar returned err = -5
Jan 18 17:50:28 homebridge kernel: [87247.886394] brcmfmac: brcmf_sdiod_probe: Failed to set F1 blocksize
Jan 18 17:50:28 homebridge kernel: [87247.886499] brcmfmac: brcmf_sdio_bus_reset: Failed to probe after sdio device reset: ret -123
Jan 18 17:50:28 homebridge kernel: [87247.887801] mmc1: card 0001 removed
Jan 18 17:50:28 homebridge kernel: [87248.012263] mmc1: queuing unknown CIS tuple 0x02 [0b 9f 10 37 b5 cb 93 26 ff 98 4a e6 44 80 00 d6 03 c9 ff 76 4d 53 ff bd 60 c7 42 ca 03 e3 50 3f 0c cf 69 d6 5f ff 12 c8 44 cd 48 cc c0 ca b3 39 00 51 4d ff 41 8c d1 a8 c8 ff 84 ff c1 cc 52 9a] (196 bytes)
Jan 18 17:50:28 homebridge kernel: [87248.014664] mmc1: queuing unknown CIS tuple 0x6e [12 81 29 22 77] (5 bytes)
Jan 18 17:50:28 homebridge kernel: [87248.075837] mmc1: queuing unknown CIS tuple 0x0c [b9 ff f5 42 31 4b 51 1a 95 69 ff 03 eb 44 7f c0 e3 24 fc 4a 30 ab 4c 71 77 40 ff 28 b3 c2 fd c0 6e 81 73 12 ff d6 29 8e 39 2a be 86 49 02 04 03 7b 22 30 13 7e 3f e9 13 e0 59 e4 42 e4 fa a4 84] (178 bytes)
Jan 18 17:50:28 homebridge kernel: [87248.096219] mmc1: queuing unknown CIS tuple 0xf0 [51 ff 32 c8 45 04 cf 43 56 47 59 a8 52 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00] (58 bytes)
I've received two test firmwares from Synaptics, numbered 1 and 2. There was no accompanying information, so I don't know if they might be a fixes or whether they are just to gather more information.
Firmware 1: https://drive.google.com/file/d/1UghnB9zZ7ydpVlYaTbp8F1M0rqEX_DZK/view?usp=share_link Firmware 2: https://drive.google.com/file/d/1P3QZDMFPR3I5wPGYWNu2SCEJli9oTCC6/view?usp=share_link
To back up the original firmware:
$ sudo cp /lib/firmware/brcm/brcmfmac43436-sdio.bin{,.orig}
To install (change test1 to test2 for the second test firmware):
$ sudo cp fw_bcmdhd_test1.bin /lib/firmware/brcm/brcmfmac43436-sdio.bin
To revert to the original firmware:
$ sudo cp /lib/firmware/brcm/brcmfmac43436-sdio.bin{.orig,}
Unfortunately it would be pointless to test those test firmwares for me I think, because I still couldnt reproduce the issuse for some reason.
I am running the test2 firmware since yesterday and so far there haven't been any firmware crashes. I'll report back in a couple of days.
@Smyl3 I understand - enjoy the lack of crashes.
@porst17 Thank you - that's sounding promising.
Unfortunately, the Wifi firmware (test2) crashed again two times over the past couple of days:
Feb 1 22:14:41 openhab kernel: [212302.000222] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
Feb 1 22:14:41 openhab kernel: [212302.046161] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046190] brcmfmac: CONSOLE: FWID 01-c43ac73d
Feb 1 22:14:41 openhab kernel: [212302.046201] brcmfmac: CONSOLE: flags 1
Feb 1 22:14:41 openhab kernel: [212302.046211] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046224] brcmfmac: CONSOLE: TRAP c(6ff7c): pc 83cf64, lr 83cf5d, sp 40ff0, cpsr 2000000c, spsr 21000000
Feb 1 22:14:41 openhab kernel: [212302.046237] brcmfmac: CONSOLE: r0 1, r1 41138, r2 2151, r3 137, r4 41138, r5 0, r6 67940
Feb 1 22:14:41 openhab kernel: [212302.046250] brcmfmac: CONSOLE: r7 5057c, r8 30, r9 1, r10 85e671, r11 41170, r12 4
Feb 1 22:14:41 openhab kernel: [212302.046261] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046272] brcmfmac: CONSOLE: sp+0 0005d1e0 00041138 00000000 0083cef5
Feb 1 22:14:41 openhab kernel: [212302.046284] brcmfmac: CONSOLE: sp+10 00041138 0005b810 00000016 0085e6c5
Feb 1 22:14:41 openhab kernel: [212302.046295] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046305] brcmfmac: CONSOLE: sp+c 0083cef5
Feb 1 22:14:41 openhab kernel: [212302.046316] brcmfmac: CONSOLE: sp+1c 0085e6c5
Feb 1 22:14:41 openhab kernel: [212302.046328] brcmfmac: CONSOLE: sp+4c 0085e7c9
Feb 1 22:14:41 openhab kernel: [212302.046211] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046224] brcmfmac: CONSOLE: TRAP c(6ff7c): pc 83cf64, lr 83cf5d, sp 40ff0, cpsr 2000000c, spsr 21000000
Feb 1 22:14:41 openhab kernel: [212302.046237] brcmfmac: CONSOLE: r0 1, r1 41138, r2 2151, r3 137, r4 41138, r5 0, r6 67940
Feb 1 22:14:41 openhab kernel: [212302.046250] brcmfmac: CONSOLE: r7 5057c, r8 30, r9 1, r10 85e671, r11 41170, r12 4
Feb 1 22:14:41 openhab kernel: [212302.046261] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046272] brcmfmac: CONSOLE: sp+0 0005d1e0 00041138 00000000 0083cef5
Feb 1 22:14:41 openhab kernel: [212302.046284] brcmfmac: CONSOLE: sp+10 00041138 0005b810 00000016 0085e6c5
Feb 1 22:14:41 openhab kernel: [212302.046295] brcmfmac: CONSOLE:
Feb 1 22:14:41 openhab kernel: [212302.046305] brcmfmac: CONSOLE: sp+c 0083cef5
Feb 1 22:14:41 openhab kernel: [212302.046316] brcmfmac: CONSOLE: sp+1c 0085e6c5
Feb 1 22:14:41 openhab kernel: [212302.046328] brcmfmac: CONSOLE: sp+4c 0085e7c9
Feb 1 22:14:41 openhab kernel: [212302.046338] brcmfmac: CONSOLE: sp+8c 0085e929
Feb 1 22:14:41 openhab kernel: [212302.046349] brcmfmac: CONSOLE: sp+98 0085e671
Feb 1 22:14:41 openhab kernel: [212302.046359] brcmfmac: CONSOLE: sp+11c 0085e515
Feb 1 22:14:41 openhab kernel: [212302.046370] brcmfmac: CONSOLE: sp+130 0085e635
Feb 1 22:14:41 openhab kernel: [212302.046380] brcmfmac: CONSOLE: sp+134 0085e671
Feb 1 22:14:41 openhab kernel: [212302.046391] brcmfmac: CONSOLE: sp+18c 0000013b
Feb 1 22:14:41 openhab kernel: [212302.046401] brcmfmac: CONSOLE: sp+1b4 0085ea59
Feb 1 22:14:41 openhab kernel: [212302.046412] brcmfmac: CONSOLE: sp+1d4 00836547
Feb 1 22:14:41 openhab kernel: [212302.046422] brcmfmac: CONSOLE: sp+22c 0085e2af
Feb 1 22:14:41 openhab kernel: [212302.046433] brcmfmac: CONSOLE: sp+25c 008390b3
Feb 1 22:14:41 openhab kernel: [212302.046443] brcmfmac: CONSOLE: sp+300 00832175
Feb 1 22:14:41 openhab kernel: [212302.046454] brcmfmac: CONSOLE: sp+304 00832165
Feb 1 22:14:41 openhab kernel: [212302.046464] brcmfmac: CONSOLE: sp+308 00832191
Feb 02 04:15:08 openhab kernel: ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: FWID 01-c43ac73d
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: flags 1
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: TRAP c(6ff7c): pc 83cf64, lr 83cf5d, sp 40ff0, cpsr 2000000c, spsr 21000000
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: r0 1, r1 41138, r2 2151, r3 137, r4 41138, r5 0, r6 67940
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: r7 50d60, r8 30, r9 1, r10 85e671, r11 41170, r12 4
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+0 0005d1e0 00041138 00000000 0083cef5
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+10 00041138 0005b810 00000016 0085e6c5
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: TRAP c(6ff7c): pc 83cf64, lr 83cf5d, sp 40ff0, cpsr 2000000c, spsr 21000000
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: r0 1, r1 41138, r2 2151, r3 137, r4 41138, r5 0, r6 67940
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: r7 50d60, r8 30, r9 1, r10 85e671, r11 41170, r12 4
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+0 0005d1e0 00041138 00000000 0083cef5
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+10 00041138 0005b810 00000016 0085e6c5
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE:
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+c 0083cef5
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+1c 0085e6c5
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+4c 0085e7c9
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+8c 0085e929
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+98 0085e671
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+11c 0085e515
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+130 0085e635
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+134 0085e671
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+18c 0000013b
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+1b4 0085ea59
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+1d4 00836547
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+22c 0085e2af
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+25c 008390b3
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+300 00832175
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+304 00832165
Feb 02 04:15:08 openhab kernel: brcmfmac: CONSOLE: sp+308 00832191
Thanks - I'll feed back your findings. Did you try test1?
Did anybody else try the new firmwares?
I was running test2 so far and switched over to the test1 firmware just now. I'll keep you updated.
Thanks again. Until we are able to reproduce the issue we are relying on people like you.
In my case test1 crashes just as often, additionally bluetooth loses connections. test2 works fine so far.
In my case test1 crashes just as often, additionally bluetooth loses connections. test2 works fine so far.
I've been running test1 now for about a week without a single crash. 🤞
Strange that others seem to observe the opposite behavior.
Indeed - it is strange.
@mati75 It would be helpful if you could find the time to capture and post a crash log from the test1 firmware.
Maybe there are more different issues in the firmware that needs different solutions.
@pelwell I will try catch crash at the weekend. I only caught that the crash occurs when dhcpv6 loses connection, but I wasn't able to save dmesg.
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wlc_join_adopt_bss, 5751
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wlc_link, 36727
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wlc_link, 36849
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: link up (wl0)
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x1 tid 0
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:22 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:28 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:33 2023] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x1 tid 4
[Fri Feb 10 18:56:34 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:34 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[Fri Feb 10 18:56:35 2023] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
ping6 google.com -c 3
PING google.com(bud02s23-in-x0e.1e100.net (2a00:1450:400d:802::200e)) 56 data bytes
--- google.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2108ms
connections down, in the same network pi0w, rpi4 working fine
All: I'm also seeing this problem, I think. My Pi Z2W is living in a Ubiquiti wireless environment. Four UAP-AC-LR access points, all advertising three different SSIDs, on SSID reserved for IoT devices and not broadcasting it's ID. The Z2W is on the IoT WiFi SSID. It can really only see two of the access points. I used Unifi controller to lock the Z2W to a single AP. This reduced spontaneous reboots from a couple of times a day to about once or twice a week, but still has not fixed the problem. Running box stock Raspian 64 bit.
Here is a syslog snippet from the latest reboot. You can see the usual FIPS logging nonsense, some trouble with the WiFi connection, then BOOM!
Feb 15 09:34:23 genmon rngd[397]: stats: bits received from HRNG source: 6200064
Feb 15 09:34:23 genmon rngd[397]: stats: bits sent to kernel pool: 6146144
Feb 15 09:34:23 genmon rngd[397]: stats: entropy added to kernel pool: 6146144
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2 successes: 310
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2 failures: 0
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Poker: 0
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Runs: 0
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Long run: 0
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Feb 15 09:34:23 genmon rngd[397]: stats: HRNG source speed: (min=906.070; avg=1070.052; max=1088.698)Kibits/s
Feb 15 09:34:23 genmon rngd[397]: stats: FIPS tests speed: (min=23.176; avg=36.915; max=43.747)Mibits/s
Feb 15 09:34:23 genmon rngd[397]: stats: Lowest ready-buffers level: 2
Feb 15 09:34:23 genmon rngd[397]: stats: Entropy starvations: 0
Feb 15 09:34:23 genmon rngd[397]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us
Feb 15 09:57:29 genmon dhcpcd[491]: wlan0: carrier lost
Feb 15 09:57:29 genmon dhcpcd[491]: wlan0: deleting route to 192.168.1.0/24
Feb 15 09:57:29 genmon dhcpcd[491]: wlan0: deleting default route via 192.168.1.1
Feb 15 09:57:36 genmon dhcpcd[491]: wlan0: carrier acquired
Feb 15 09:57:36 genmon dhcpcd[491]: wlan0: IAID 01:44:3d:a1
Feb 15 09:57:36 genmon dhcpcd[491]: wlan0: probing address 192.168.1.62/24
Feb 15 09:57:37 genmon dhcpcd[491]: wlan0: soliciting an IPv6 router
Feb 15 09:57:41 genmon dhcpcd[491]: wlan0: using static address 192.168.1.62/24
Feb 15 09:57:41 genmon dhcpcd[491]: wlan0: adding route to 192.168.1.0/24
Feb 15 09:57:41 genmon dhcpcd[491]: wlan0: adding default route via 192.168.1.1
Feb 15 09:57:49 genmon dhcpcd[491]: wlan0: carrier lost
Feb 15 09:57:49 genmon dhcpcd[491]: wlan0: deleting route to 192.168.1.0/24
Feb 15 09:57:49 genmon dhcpcd[491]: wlan0: deleting default route via 192.168.1.1
Feb 15 09:57:51 genmon dhcpcd[491]: wlan0: carrier acquired
Feb 15 09:57:51 genmon dhcpcd[491]: wlan0: IAID 01:44:3d:a1
Feb 15 09:57:51 genmon dhcpcd[491]: wlan0: probing address 192.168.1.62/24
Feb 15 09:57:51 genmon dhcpcd[491]: wlan0: soliciting an IPv6 router
Feb 15 09:57:56 genmon dhcpcd[491]: wlan0: using static address 192.168.1.62/24
Feb 15 09:57:56 genmon dhcpcd[491]: wlan0: adding route to 192.168.1.0/24
Feb 15 09:57:56 genmon dhcpcd[491]: wlan0: adding default route via 192.168.1.1
Feb 15 10:02:44 genmon dhcpcd[491]: wlan0: carrier lost
Feb 15 10:02:44 genmon dhcpcd[491]: wlan0: deleting route to 192.168.1.0/24
Feb 15 10:02:44 genmon dhcpcd[491]: wlan0: deleting default route via 192.168.1.1
Feb 15 10:02:46 genmon dhcpcd[491]: wlan0: carrier acquired
Feb 15 10:02:46 genmon dhcpcd[491]: wlan0: IAID 01:44:3d:a1
Feb 15 10:02:46 genmon dhcpcd[491]: wlan0: probing address 192.168.1.62/24
Feb 15 10:02:47 genmon dhcpcd[491]: wlan0: soliciting an IPv6 router
Feb 15 10:02:51 genmon dhcpcd[491]: wlan0: using static address 192.168.1.62/24
Feb 15 10:02:51 genmon dhcpcd[491]: wlan0: adding route to 192.168.1.0/24
Feb 15 10:02:51 genmon dhcpcd[491]: wlan0: adding default route via 192.168.1.1
Feb 15 10:17:01 genmon CRON[182485]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 15 10:34:23 genmon rngd[397]: stats: bits received from HRNG source: 6220064
Feb 15 10:34:23 genmon rngd[397]: stats: bits sent to kernel pool: 6176384
Feb 15 10:34:23 genmon rngd[397]: stats: entropy added to kernel pool: 6176384
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2 successes: 311
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2 failures: 0
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Poker: 0
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Runs: 0
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Long run: 0
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Feb 15 10:34:23 genmon rngd[397]: stats: HRNG source speed: (min=906.070; avg=1070.062; max=1088.698)Kibits/s
Feb 15 10:34:23 genmon rngd[397]: stats: FIPS tests speed: (min=23.176; avg=36.866; max=43.747)Mibits/s
Feb 15 10:34:23 genmon rngd[397]: stats: Lowest ready-buffers level: 2
Feb 15 10:34:23 genmon rngd[397]: stats: Entropy starvations: 0
Feb 15 10:34:23 genmon rngd[397]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us
Feb 15 10:17:07 genmon fake-hwclock[132]: Wed 15 Feb 15:17:01 UTC 2023
Feb 15 10:17:07 genmon systemd-fsck[146]: e2fsck 1.46.2 (28-Feb-2021)
Feb 15 10:17:07 genmon systemd-fsck[146]: rootfs: clean, 56189/473280 files, 668565/1873920 blocks
Feb 15 10:17:07 genmon systemd[1]: Finished Remount Root and Kernel File Systems.
Feb 15 10:17:07 genmon systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Feb 15 10:17:07 genmon systemd[1]: Starting Flush Journal to Persistent Storage...
Feb 15 10:17:07 genmon systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
Feb 15 10:17:07 genmon kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
Sorry, Phil, but I had no way of knowing you and I were editing in parallel. Your last edit truncated the log. It appears fine now. I could not get it to show properly in code tags for some reason.
The WiFi firmware crashing (if that's what's happening to you - you don't appear to have added brcmfmac.debug=0x100000 to /boot/cmdline.txt as requested above) shouldn't cause your Pi to reboot.
How are you powering the Pi and its peripherals?
I made the change and rebooted as requested. Now we need to wait for a spontaneous reboot. Where should I look for the relevant log entries?
The Z2W is powered by a large, commercial, 5VDC power supply that can provide 5A. The only peripheral the Z2W is powering is a single RS232-TTL level translator IC on the 3.3VDC bus which consumes well under 10ma. The Z2W never reports undervoltage or throttling. The SD card has been replaced once as a possible source of the problem.
The additional log entries will go into the kernel log, which should appear in the main system log as above.
So, Im still using the original fw. that still didnt crash but I had a weird wifi connection issue today until my script was restarted brcmfmac kernel driver at 14:20. Dont know is it related to this issue or not.
The issue started with a DHCPREQUEST and I see a lot of
wpa_supplicant[589307]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=MAC status_code=16 messages in syslog....
kern.log.23.02.18.1415.1420.log syslog.23.02.18.1415.1420.log