linux icon indicating copy to clipboard operation
linux copied to clipboard
verified publisher icon raspberrypi

Pi5 failed to contact RP1 firmware

Open llllllllqq opened this issue 11 months ago • 33 comments

Describe the bug

dmesg says:

... rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware [ 3.957885] rp1-pio: probe of 1f00178000.pio failed with error -2 ...

Although nothing strange occurred. The OS still works fine without reboot.

Steps to reproduce the behaviour

It randomly occurs after sudo reboot --halt and then manually press the on-board power button to power on (with default eeprom config).

~~It also randomly occurs when use raspi-config to change power off behavior to "full power off" and then reboot/poweroff.~~ (Sorry I cannot remember whether it is reboot or poweroff)

Update: It also occurs after a suddenly plug-out the usb-c power cable and then plug-in back. ~~It seems only poweroff-related behaviors will trigger this bug. I tested 30 times reboot, no bugs occurred.~~ Update2: reboot can indeed also trigger this bug, only less frequent.

Device (s)

Raspberry Pi 5

System

Raspberry Pi reference 2024-11-19 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 891df1e21ed2b6099a2e6a13e26c91dea44b34d4, stage4

2025/01/22 00:16:51 Copyright (c) 2012 Broadcom version a7753063 (release) (embedded)

Linux rpi5 6.6.74+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64 GNU/Linux

Logs

1.txt

Additional context

Can it be safely ignored?

llllllllqq avatar Feb 01 '25 07:02 llllllllqq

That's interesting - it looks like a dependency/ordering issue, and not one I've seen before:

[    3.922665] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    3.922671] rp1-pio: probe of 1f00178000.pio failed with error -2
...
[    3.932284] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55

I get:

[   13.445656] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[   13.550428] rp1-pio 1f00178000.pio: Created instance as pio0

The code for communicating with the firmware exists in the firmware module, so I thought that would enforce an ordering. Some investigation is required.

pelwell avatar Feb 03 '25 10:02 pelwell

That's interesting - it looks like a dependency/ordering issue, and not one I've seen before:

[    3.922665] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    3.922671] rp1-pio: probe of 1f00178000.pio failed with error -2
...
[    3.932284] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55

I get:

[   13.445656] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[   13.550428] rp1-pio 1f00178000.pio: Created instance as pio0

The code for communicating with the firmware exists in the firmware module, so I thought that would enforce an ordering. Some investigation is required.

Thanks for your reply. Here is my dmesg when the rp1&pio works fine. I hope it can help to investigate this issue.

normal.txt

llllllllqq avatar Feb 03 '25 10:02 llllllllqq

Yes - in the working case the order is reversed:

[    3.955929] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[    3.961971] vc4-drm axi:gpu: bound 107c580000.hvs (ops vc4_hvs_ops [vc4])
[    3.967397] rp1-pio 1f00178000.pio: Created instance as pio0

pelwell avatar Feb 03 '25 10:02 pelwell

[ 5.557565] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware [ 5.557571] rp1-pio: probe of 1f00178000.pio failed with error -2 [ 5.558534] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55

Seeing that on reboot as well /custom kernel on a Pi5 DUT, trying to get an overlay to work)

Linux Pi5uSDDUT 6.6.74-v8-16k+ #1 SMP PREEMPT Sun Feb 2 10:41:23 CET 2025 aarch64 GNU/Linux Revision : c04170 Serial : cb30eb1fd7e7e672 Model : Raspberry Pi 5 Model B Rev 1.0

aBUGSworstnightmare-rpi avatar Feb 03 '25 14:02 aBUGSworstnightmare-rpi

It's really strange. I can't get it to fail in that way without hacks - even with a massive delay in the firmware driver, the pio driver waits. Anyway, I'm just testing a fix.

pelwell avatar Feb 03 '25 14:02 pelwell

seems quite random. Seen it frequently on reboots (running 'sudo reboot' on CLI)

aBUGSworstnightmare-rpi avatar Feb 03 '25 15:02 aBUGSworstnightmare-rpi

is there a way to fix this? I keep getting this error and I can't boot the OS, should I be worried that I will loose data on my raspberry?

mayo228 avatar Feb 03 '25 15:02 mayo228

I don't understand why this affecting some people while I can't reproduce it without hacking the driver, but regardless, https://github.com/raspberrypi/linux/pull/6645 should fix the issue. After about 40 minutes you should be able to install a trial kernel with sudo rpi-update pulls/6645, after first backing up any important data.

pelwell avatar Feb 03 '25 15:02 pelwell

And no, you shouldn't worry - it would only affect you if you wanted to use piolib to drive the GPIOs.

pelwell avatar Feb 03 '25 15:02 pelwell

I have no idea if and how below is related but I just wanted to state that I'm not making use of piolib for accessing GPIO's.

[   13.678330] ieee80211 phy0: brcmf_p2p_send_action_frame: Unknown Frame: category 0xa, action 0x8
[   15.374753] Bluetooth: RFCOMM TTY layer initialized
[   15.374765] Bluetooth: RFCOMM socket layer initialized
[   15.374772] Bluetooth: RFCOMM ver 1.11
[   27.660651] brcmfmac: brcmf_set_channel: set chanspec 0xd022 fail, reason -52
[   27.768681] brcmfmac: brcmf_set_channel: set chanspec 0xd026 fail, reason -52
[   27.872656] brcmfmac: brcmf_set_channel: set chanspec 0xd02a fail, reason -52
[   27.976661] brcmfmac: brcmf_set_channel: set chanspec 0xd02e fail, reason -52
[   29.636663] brcmfmac: brcmf_set_channel: set chanspec 0xd090 fail, reason -52
[   29.636875] brcmfmac: brcmf_set_channel: set chanspec 0xd095 fail, reason -52
[   29.637196] brcmfmac: brcmf_set_channel: set chanspec 0xd099 fail, reason -52
[   29.637400] brcmfmac: brcmf_set_channel: set chanspec 0xd09d fail, reason -52
[   29.637594] brcmfmac: brcmf_set_channel: set chanspec 0xd0a1 fail, reason -52
[   29.637791] brcmfmac: brcmf_set_channel: set chanspec 0xd0a5 fail, reason -52

aBUGSworstnightmare-rpi avatar Feb 03 '25 16:02 aBUGSworstnightmare-rpi

I have no idea if and how below is related

You should know that WiFi and Bluetooth information and warning messages are unrelated.

pelwell avatar Feb 03 '25 16:02 pelwell

It seems after applying sudo rpi-update pulls/6645, this issue disappeared. (I tested about 10 times reboot/halt/suddenly poweroff). Thanks for your work! If anything strange found, I'll post here.

BTW, can I safely stay at this firmware and wait for the next stable release offered by apt? Or do I need a sudo apt install --reinstall raspi-firmware to restore first before the next stable release?

llllllllqq avatar Feb 04 '25 00:02 llllllllqq

You should be fine with this kernel until the next proper release - I'm not aware of any new issues.

pelwell avatar Feb 04 '25 09:02 pelwell

I can confirm the error happens on my PI 5 too:

[    0.692347] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    0.692370] rp1-pio: probe of 1f00178000.pio failed with error -2

I am running the latest firmware:

BOOTLOADER: up to date
   CURRENT: Wed 22 Jan 00:16:51 UTC 2025 (1737505011)
    LATEST: Wed 22 Jan 00:16:51 UTC 2025 (1737505011)
   RELEASE: latest (/usr/lib/firmware/raspberrypi/bootloader-2712/latest)
            Use raspi-config to change the release.

randomMesh avatar Feb 07 '25 18:02 randomMesh

A week ago I also started receiving these errors during bootup:

rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware

At the same time I also started getting protocol errors for the wireless mouse and wireless keyboard logitech-hidpp-device - hidpp_root_get_ protocol_version: received protocol error 0x08 (Although - both devices continued to work appropriately.)

Changing to wired devices there are no mouse or keyboard errors but still have the rp1-pio error

I have performed the "sudo rpi-update pulls/6645" and no change was observed.

helpful-emu57 avatar Feb 09 '25 03:02 helpful-emu57

What does rpi-eeprom-update show?

pelwell avatar Feb 09 '25 08:02 pelwell

How do you do "sudo rpi-update pulls/6645" if you can't boot into it?

My new pi-500 seems to be bricked with this error. I just tried to install moonlight-qt and performed the "sudo reboot" and can't get it to boot up.

nickongit avatar Feb 09 '25 23:02 nickongit

My new pi-500 seems to be bricked with this error. I just tried to install moonlight-qt and performed the "sudo reboot" and can't get it to boot up.

From the detailed description of your problem I'm going to guess that:

  1. You had a working Pi 500.
  2. You installed some new software.
  3. It no longer boots.
  4. The only error message you can see on screen is the one about the RP1 firmware.

As I've explained above, the only effect of that error is to make PIO unavailable - it will not cause the boot to fail. I suggest you edit cmdline.txt in another computer, replacing quiet with ignore_loglevel, then try again. You should see much more output, and that may make it clearer what is going wrong.

pelwell avatar Feb 10 '25 10:02 pelwell

RE: follow-up 3 days ago - Boot RP1 Firmware Error

  Excerpt From Boot Log
  [    0.858841] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
  [    0.858866] rp1-pio: probe of 1f00178000.pio failed with error -2
  
  Is location 1f00178000 reserved for particular hardware? 
  What is an "rp1-pio error -2" error ?

RE: Wireless Device Errors The protocol error is gone when using original wireless mouse, keyboard and dongle (would similar make/model with different dongle cause error?)

helpful-emu57 avatar Feb 12 '25 01:02 helpful-emu57

And my response three days ago was:

What does rpi-eeprom-update show?

You will see that harmless error until you update the EEPROM firmware.

pelwell avatar Feb 12 '25 07:02 pelwell

Apologies, thank you for the response, i meant to reply yesterday when i had a chance to fix it.

The root cause of no boot was a cron entry to start moonlight-qt @reboot.

Strangely i accidentally found the solution to fixing it by tryinf another monitor, which resulted in the pi booting before i plugged in the monitor. Then i could plug in monitor and take action removing the cron entry.

As you suggested the error is always there and was not the cause of the boot problem, just the last entry visible on the boot screen.

nickongit avatar Feb 12 '25 16:02 nickongit

I think it might be best if the error was demoted to a warning.

pelwell avatar Feb 12 '25 16:02 pelwell

I re-installed rpi-eeprom [Wed 12 Feb 2025 10:51:52 AM UTC (1739357512)] and then [rpi-update pulls/6645] again and this time it worked.

helpful-emu57 avatar Feb 20 '25 18:02 helpful-emu57

Hello.

How did you do that?

Erwin40 avatar Feb 21 '25 09:02 Erwin40

Hello just want to report I have the same error and unfortunately even after re-install of rpi-eeprom it is still kicking the error.

just to be sure it installed okay, report the output of vcgencmd bootloader_version and uname -a.

popcornmix avatar Feb 21 '25 13:02 popcornmix

In the Terminal Run "rpi-eeprom-update" and report output

rpi-eeprom-update

helpful-emu57 avatar Feb 21 '25 16:02 helpful-emu57

Off-topic discussion deleted.

pelwell avatar Feb 24 '25 10:02 pelwell

I also had the error message “failed to contact rp1 firmware“. So I installed a new sd card as a test. This error message didn't appear with this new sd card, even though it uses the same kernel. When comparing the /boot/firmware directory, I noticed that the initramfs files on the older card were ~17 MB in size and the two new ones were 12.7 MB. After copying the two files from the new sd card to the old one, the error disappeared.

ozapft avatar Mar 02 '25 20:03 ozapft

I'm also seeing this error sometimes, but only on Pi5 with NVMe (non-boot) drive attached.

RPiOS: 12.9 Bookworm 64-bit | Linux: 6.6.74 64-bit (built with Pi-Gen)
ii  raspi-firmware   1:1.20250305-1  all  Raspberry Pi family GPU firmware and bootloaders

I have 2 such Pi5's and one symptom is that sometimes reboot or power off does not complete and the Pi never reaches the halt state (red LED). Two other Pi5's w/o NVMe drive are fine.

Pi5 with NVMe drive

pi@moode9:~ $ dmesg | grep rp1
[    0.805353] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    0.823933] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    2.284970] rp1 0001:01:00.0: bar0 len 0x4000, start 0x1f00410000, end 0x1f00413fff, flags, 0x40200
[    2.294054] rp1 0001:01:00.0: bar1 len 0x400000, start 0x1f00000000, end 0x1f003fffff, flags, 0x40200
[    2.303317] rp1 0001:01:00.0: enabling device (0000 -> 0002)
[    2.309638] rp1 0001:01:00.0: chip_id 0x20001927
[    2.318624] genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 106
[    5.742889] rp1-firmware: probe of rp1_firmware failed with error -110
[    5.749497] rp1-pio 1f00178000.pio: error -ENOENT: failed to contact RP1 firmware
[    5.749505] rp1-pio: probe of 1f00178000.pio failed with error -2
pi@moode9:~ $

Pi5 w/o NVMe drive

pi@moode:~ $ dmesg | grep rp1
[    1.015138] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    1.033523] /axi/pcie@120000/rp1: Fixed dependency cycle(s) with /axi/pcie@120000/rp1
[    2.137633] rp1 0000:01:00.0: bar0 len 0x4000, start 0x1f00410000, end 0x1f00413fff, flags, 0x40200
[    2.146719] rp1 0000:01:00.0: bar1 len 0x400000, start 0x1f00000000, end 0x1f003fffff, flags, 0x40200
[    2.155981] rp1 0000:01:00.0: enabling device (0000 -> 0002)
[    2.162270] rp1 0000:01:00.0: chip_id 0x20001927
[    2.170959] genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 100
[    5.009561] rp1-firmware rp1_firmware: RP1 Firmware version eb39cfd516f8c90628aa9d91f52370aade5d0a55
[    5.237119] rp1-pio 1f00178000.pio: Created instance as pio0
pi@moode:~ $

PCIe controller: Pimoroni NVMe Base for Raspberry Pi 5 https://www.adafruit.com/product/5845?gad_source=1&gbraid=0AAAAADx9JvT2Uc1scGbd8pfLnNHlqrIyp&gclid=EAIaIQobChMIz8PE47P6iwMVo15_AB2ZfhpaEAQYASABEgKY__D_BwE

NVMe stick: Crucial P3 1TB PCIe Gen3 3D NAND NVMe M.2 SSD https://www.bhphotovideo.com/c/product/1710645-REG/crucial_ct1000p3ssd8_1tb_p3_nvme_pcie.html/?ap=y&ap=y&smp=y&smp=y&store=420&lsft=BI%3A5451&gad_source=1&gbraid=0AAAAAD7yMh0CnuDG9lQosL1qx1hNsryh6&gclid=EAIaIQobChMIiaSZwLT6iwMVHDStBh17LgH0EAQYASABEgLYhvD_BwE

moodeaudio avatar Mar 08 '25 11:03 moodeaudio

It's possible that the presence of the NVME drive changes the timing of the boot procedure. Between 6.6.74 and 6.6.75 the RP1 PIO and firmware drivers were improved to cope with arbitrary startup order. I can't tell whether or not the kernel you have includes that patch, but you can ensure that it does by running sudo rpi-update 1705add.

pelwell avatar Mar 08 '25 20:03 pelwell