build icon indicating copy to clipboard operation
build copied to clipboard
verified publisher icon armbian

sunxi: `reboot` does not perform a reboot

Open djurny opened this issue 7 months ago • 5 comments

Seems that a recent rolling update has made my sunxi Orange Pi zero boards stuck at reboot. Kernel shuts down fine, no errors, prints out "Restarting system" or "Rebooting system" but it seems to just get stuck. Magic SysRq not working so kernel has left the building.

[84345.585698] systemd-shutdown[1]: Using hardware watchdog 'sunxi-wdt', version 0, device /dev/watchdog0
[84345.595240] systemd-shutdown[1]: Failed to set watchdog hardware timeout to 10min: Invalid argument
[84345.684052] systemd-shutdown[1]: Syncing filesystems and block devices.
[84345.755564] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[84345.780255] systemd-journald[228]: Received SIGTERM from PID 1 (systemd-shutdow).
[84345.877323] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[84345.902339] systemd-shutdown[1]: Unmounting file systems.
[84345.912897] (sd-umount)[11817]: Unmounting '/tmp'.
[84345.922920] (sd-remount)[11818]: Remounting '/' read-only with options 'background_gc=on,nogc_merge,discard,discard_unit=block,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,barrier,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,checkpoint_merge,fsync_mode=posix,memory=normal,errors=continue'.
[84346.040634] systemd-shutdown[1]: All filesystems unmounted.
[84346.046333] systemd-shutdown[1]: Deactivating swaps.
[84346.051504] systemd-shutdown[1]: All swaps deactivated.
[84346.056778] systemd-shutdown[1]: Detaching loop devices.
[84346.066741] systemd-shutdown[1]: All loop devices detached.
[84346.072373] systemd-shutdown[1]: Stopping MD devices.
[84346.078030] systemd-shutdown[1]: All MD devices stopped.
[84346.083398] systemd-shutdown[1]: Detaching DM devices.
[84346.089090] systemd-shutdown[1]: All DM devices detached.
[84346.094538] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[84346.112307] systemd-shutdown[1]: Syncing filesystems and block devices.
[84346.119273] systemd-shutdown[1]: Rebooting.
[84346.164738] reboot: Restarting system

No further output or activity observed after this. Board does not actually reboot/reset. It requires a power cycle to start again.

Could be that the board just halts or powers down, but not quite sure how to determine that with for example a multimeter.

Will update the description here after I found out what rolling release version caused this to happen.

Groetjes,

djurny avatar May 15 '25 18:05 djurny

Jira ticket: AR-2684

github-actions[bot] avatar May 15 '25 18:05 github-actions[bot]

Downgraded from 25.2.3 to 24.11.1 seems to resolve the issue (includes tio timestamps):

[19:49:45] [   87.616053] systemd-shutdown[1]: Stopping MD devices.
[19:49:45] [   87.621767] systemd-shutdown[1]: All MD devices stopped.
[19:49:45] [   87.627230] systemd-shutdown[1]: Detaching DM devices.
[19:49:45] [   87.632944] systemd-shutdown[1]: All DM devices detached.
[19:49:45] [   87.638393] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[19:49:45] [   87.656493] systemd-shutdown[1]: Syncing filesystems and block devices.
[19:49:45] [   87.663469] systemd-shutdown[1]: Rebooting.
[19:49:45] [   87.720599] reboot: Restarting system

[19:49:46] U-Boot SPL 2024.01-armbian (Feb 23 2024 - 10:47:39 +0000)
[19:49:46] DRAM: 256 MiB
[19:49:46] Trying to boot from MMC1
[19:49:46] ns16550_serial serial@1c28000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19


[19:49:46] U-Boot 2024.01-armbian (Feb 23 2024 - 10:47:39 +0000) Allwinner Technology

[DOWNGRADE] armbian-bsp-cli-orangepizero-current:armhf 25.2.3 -> 24.11.1
[DOWNGRADE] armbian-firmware:armhf 25.2.3 -> 24.11.1
[DOWNGRADE] linux-dtb-current-sunxi:armhf 25.2.3 -> 24.11.1
[DOWNGRADE] linux-image-current-sunxi:armhf 25.2.3 -> 24.11.1
[UPGRADE] linux-u-boot-orangepizero-current:armhf 24.2.1 -> 24.11.1

Note that I at first thought it was due to U-Boot, so I tried with built U-Boots and downgraded linux-u-boot-orangepizero-current packages.

Will dive into the diffs to see if I can find the culprit.

Grt,

djurny avatar May 16 '25 02:05 djurny

In 25.2.2 reboot also works.

========================================
...
[DOWNGRADE] armbian-bsp-cli-orangepizero-current:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] armbian-firmware:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] linux-dtb-current-sunxi:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] linux-image-current-sunxi:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] linux-u-boot-orangepizero-current:armhf 25.08.0-trunk -> 25.2.2
========================================

djurny avatar May 16 '25 22:05 djurny

Seems like issue introduced when moving from 6.6.75-current-sunxi to 6.12.23-current-sunxi

djurny avatar May 21 '25 00:05 djurny

Downgrading u-boot package only replaces binaries available for flashing. Though flashing does not happen automatically since it is rarely necessary to actually do that. sunxi family is very heavily patched so regressions are likely. Finding the cause though can be quite a challenge.

EvilOlaf avatar May 21 '25 03:05 EvilOlaf

Possible solution: https://forum.armbian.com/topic/22810-trying-pcduino2-on-a-newer-kernel/

igorpecovnik avatar Jun 29 '25 10:06 igorpecovnik

I tested both 6.12.3x and 6.15.x on the Orange Pi One, and both can reboot normally. However, on the Orange Pi Zero, using 6.12.35-current-sunxi fails to reboot properly. $ sudo shutdown -r now [ 305.251956] systemd-shutdown[1]: Failed to set watchdog hardware timeout to 10min: Invalid argument [ 305.585603] reboot: Restarting system`

Issuing the command echo 1 > /dev/watchdog allows the system to reboot normally.

After downgrading the kernel to 6.6.75-legacy-sunxi, it can reboot normally. The BOARDFAMILY of both the Orange Pi Zero and Orange Pi One is "sun8i".

d93921012 avatar Jul 25 '25 08:07 d93921012

I commented out CRUSTCONFIG="orangepi_zero_defconfig" in config/boards/orangepizero.csc, and the built image can reboot normally. Tested with RELEASE=noble on both BRANCH=current and BRANCH=edge, and both work properly.

d93921012 avatar Aug 05 '25 06:08 d93921012

I commented out CRUSTCONFIG="orangepi_zero_defconfig" in config/boards/orangepizero.csc, and the built image can reboot normally.

Then we probably lost https://github.com/crust-firmware/crust?tab=readme-ov-file

It would be nice to figure out when and why this stopped working.

igorpecovnik avatar Aug 05 '25 13:08 igorpecovnik

I installed Debian 13 Trixie on my Orange Pi Zero v1 for test

Linux debian 6.12.38+deb13-armmp #1 SMP Debian 6.12.38-1 (2025-07-16) armv7l

and reboot working

but Armbian with kernel 6.12.35 has a problem with reboot (make like poweroff ) and not restart computer

Next I updated Debian distribution which is linux-image-6.12.41+deb13-armmp

Linux debian 6.12.41+deb13-armmp #1 SMP Debian 6.12.41-1 (2025-08-12) armv7l

and reboot working

I see that Debian use u-boot v2025.01

sp2ong avatar Aug 15 '25 09:08 sp2ong

Sorry, but it looks like used U-Boot 2025.07

Image

sp2ong avatar Aug 15 '25 10:08 sp2ong

I am sure reboot will work again if we drop APM (advanced powered management) aka CRUST firmware, which is almost certainly not present in Debian images. Since making Crust operational is too much work, its probably best to drop it.

igorpecovnik avatar Aug 15 '25 14:08 igorpecovnik

OK, I understand this requires kernel compilation. So if it's available via apt update/apt upgrade, we can quickly check it out.

sp2ong avatar Aug 15 '25 14:08 sp2ong

This is u-boot add-on which is packaged, but it updating goes via armbian-config - update bootloader after new package is installed.

igorpecovnik avatar Aug 15 '25 14:08 igorpecovnik

You can generate u-boot from that PR, install u-boot package manually and update via armbian-config ... but its not needed as it was verified by @d93921012

igorpecovnik avatar Aug 15 '25 14:08 igorpecovnik

This is u-boot add-on which is packaged, but it updating goes via armbian-config - update bootloader after new package is installed.

Ok I will wait for the new package and update via armbin-install

sp2ong avatar Aug 15 '25 15:08 sp2ong

will this be an update U-Boot to 2025.07 ? like used by Debian for OZPI v1 ?

sp2ong avatar Aug 15 '25 15:08 sp2ong

will this be an update U-Boot to 2025.07

That usually brings nothing or problems, while Debian just provide automated untested builds.

igorpecovnik avatar Aug 15 '25 15:08 igorpecovnik

closed via #8503

leggewie avatar Aug 15 '25 16:08 leggewie

I try get an update and upgrade on my OZPI v1 to check reboot but the whole day is

Err:2 http://apt.armbian.com bullseye/main armhf armbian-zsh all 25.8.1 File has unexpected size (12001044 != 11982476). Mirror sync in progress? [IP: 46.102.157.202 80] Get:3 http://armbian.systemonachip.net/apt bullseye/main armhf linux-dtb-current-sunxi armhf 25.8.1 [211 kB] Err:3 http://armbian.systemonachip.net/apt bullseye/main armhf linux-dtb-current-sunxi armhf 25.8.1 File has unexpected size (210992 != 210996). Mirror sync in progress? [IP: 46.102.157.202 80] Hashes of expected file:

  • SHA512:857b35e08cada5a1d950ec0b9a32e6d1a11eca1b6b1d6a0d107556a8f2b54e721f031c21c8d36335694a0c72e659615af9ff7dc56a8803c7a809937a09396f8d
  • SHA256:06577b012018e5790ffa75d7ac4f224bee8afecdec92ec7d59ea60e201c31a1c
  • SHA1:a1d3cbd97248985b8a6d06a4f5f109a0759c66ef [weak]
  • MD5Sum:4e12ecbeae20246b2ecb1714b4bb41b8 [weak]
  • Filesize:210996 [weak]

sp2ong avatar Aug 16 '25 15:08 sp2ong

temporally issue. unrelated.

"Mirror sync in progress?" yes

igorpecovnik avatar Aug 16 '25 15:08 igorpecovnik

OK,

On another OZPI v1 with bookworm problem, connect to fi.mirror.armbian.de so I suppose tomorrow all will be back to normal :-)

Hit:1 http://deb.debian.org/debian bookworm InRelease Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB] Hit:3 http://security.debian.org bookworm-security InRelease Get:4 http://deb.debian.org/debian bookworm-backports InRelease [59.4 kB] Hit:6 https://github.armbian.com/configng stable InRelease Ign:5 http://fi.mirror.armbian.de/beta bookworm InRelease
Ign:5 http://fi.mirror.armbian.de/beta bookworm InRelease Ign:5 http://fi.mirror.armbian.de/beta bookworm InRelease Err:5 http://fi.mirror.armbian.de/beta bookworm InRelease Could not connect to fi.mirror.armbian.de:80 (65.21.120.247), connection timed out Fetched 115 kB in 38s (3,015 B/s) Reading package lists... Done W: Failed to fetch http://beta.armbian.com/dists/bookworm/InRelease Could not connect to fi.mirror.armbian.de:80 (65.21.120.247), connection timed out W: Some index files failed to download. They have been ignored, or old ones used instead.

sp2ong avatar Aug 16 '25 15:08 sp2ong

fi.mirror.armbian.de

Just checked the machine, runs normal. Slightly slower than usual but that's it. https://status.armbian.de/endpoints/2-mirror_fi-mirror

EvilOlaf avatar Aug 16 '25 15:08 EvilOlaf

At current

The following packages will be upgraded: armbian-bsp-cli-orangepizero-current armbian-firmware base-files linux-dtb-current-sunxi linux-image-current-sunxi linux-u-boot-orangepizero-current 6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 139 MB of archives. After this operation, 18.4 kB disk space will be freed. Do you want to continue? [Y/n] Ign:1 http://beta.armbian.com bookworm/bookworm-utils armhf base-files armhf 25.11.0-trunk.31-12.4+deb12u11-bookworm Ign:2 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-bsp-cli-orangepizero-current armhf 25.11.0-trunk.31 Ign:3 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-firmware all 25.11.0-trunk.31 Ign:4 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-dtb-current-sunxi armhf 25.11.0-trunk.31 Ign:5 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-image-current-sunxi armhf 25.11.0-trunk.31 Ign:6 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-u-boot-orangepizero-current armhf 25.11.0-trunk.31 Ign:1 http://beta.armbian.com bookworm/bookworm-utils armhf base-files armhf 25.11.0-trunk.31-12.4+deb12u11-bookworm Ign:2 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-bsp-cli-orangepizero-current armhf 25.11.0-trunk.31 Ign:3 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-firmware all 25.11.0-trunk.31 Ign:4 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-dtb-current-sunxi armhf 25.11.0-trunk.31 Ign:5 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-image-current-sunxi armhf 25.11.0-trunk.31 Ign:6 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-u-boot-orangepizero-current armhf 25.11.0-trunk.31 Ign:1 http://beta.armbian.com bookworm/bookworm-utils armhf base-files armhf 25.11.0-trunk.31-12.4+deb12u11-bookworm Ign:2 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-bsp-cli-orangepizero-current armhf 25.11.0-trunk.31 Ign:3 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-firmware all 25.11.0-trunk.31 Ign:4 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-dtb-current-sunxi armhf 25.11.0-trunk.31 Ign:5 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-image-current-sunxi armhf 25.11.0-trunk.31 Ign:6 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-u-boot-orangepizero-current armhf 25.11.0-trunk.31 Err:1 http://beta.armbian.com bookworm/bookworm-utils armhf base-files armhf 25.11.0-trunk.31-12.4+deb12u11-bookworm Could not connect to fi.mirror.armbian.de:80 (65.21.120.247), connection timed out Err:2 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-bsp-cli-orangepizero-current armhf 25.11.0-trunk.31 Unable to connect to fi.mirror.armbian.de:http: Ign:3 http://fi.mirror.armbian.de/beta bookworm/main armhf armbian-firmware all 25.11.0-trunk.31 Err:3 http://beta.armbian.com bookworm/main armhf armbian-firmware all 25.11.0-trunk.31 Unable to connect to fi.mirror.armbian.de:http: Err:4 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-dtb-current-sunxi armhf 25.11.0-trunk.31 Unable to connect to fi.mirror.armbian.de:http: Err:5 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-image-current-sunxi armhf 25.11.0-trunk.31 Unable to connect to fi.mirror.armbian.de:http: Err:6 http://fi.mirror.armbian.de/beta bookworm/main armhf linux-u-boot-orangepizero-current armhf 25.11.0-trunk.31 Unable to connect to fi.mirror.armbian.de:http: E: Failed to fetch http://fi.mirror.armbian.de/beta/pool/bookworm-utils/b/base-files/base-files_25.11.0-trunk.31-12.4+deb12u11-bookworm_armhf__1-bookworm-1armbian1-Baa17-U12.4--deb12u11-R55fa.deb Could not connect to fi.mirror.armbian.de:80 (65.21.120.247), connection timed out E: Failed to fetch http://fi.mirror.armbian.de/beta/pool/main/a/armbian-bsp-cli-orangepizero-current/armbian-bsp-cli-orangepizero-current_25.11.0-trunk.31_armhf__1-PCee86-V6766-H1893-B1e5d-Rf36a.deb Unable to connect to fi.mirror.armbian.de:http: E: Failed to fetch http://beta.armbian.com/pool/main/a/armbian-firmware/armbian-firmware_25.11.0-trunk.31_all__1-SA105f-B6c7f-R448a.deb Unable to connect to fi.mirror.armbian.de:http: E: Failed to fetch http://fi.mirror.armbian.de/beta/pool/main/l/linux-dtb-current-sunxi/linux-dtb-current-sunxi_25.11.0-trunk.31_armhf__6.12.35-S783c-D0b5d-P0ef8-C34a1Hf8af-HK01ba-V014b-Bd200-R448a.deb Unable to connect to fi.mirror.armbian.de:http: E: Failed to fetch http://fi.mirror.armbian.de/beta/pool/main/l/linux-6.12.35/linux-image-current-sunxi_25.11.0-trunk.31_armhf__6.12.35-S783c-D0b5d-P0ef8-C34a1Hf8af-HK01ba-V014b-Bd200-R448a.deb Unable to connect to fi.mirror.armbian.de:http: E: Failed to fetch http://fi.mirror.armbian.de/beta/pool/main/l/linux-u-boot-orangepizero-current/linux-u-boot-orangepizero-current_25.11.0-trunk.31_armhf__2024.01-S866c-P7738-Ha5c2-Vd711-Bb703-R448a.deb Unable to connect to fi.mirror.armbian.de:http: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

sp2ong avatar Aug 16 '25 16:08 sp2ong

After a few attempts, I managed to update.

I can confirm that rebooting works on the Orange Pi Zero on

Armbian 25.11.0-trunk.31 bookworm

sp2ong avatar Aug 16 '25 17:08 sp2ong

Must have been a routing issue or something. Just check for a random file from your logs which failed and folks were successful grabbing.

$ cat mirror_access.log|grep "base-files_25.11.0-trunk.31-12.4+deb12u11-bookworm_armhf__1-bookworm-1armbian1-Baa17-U12.4--deb12u11-R55fa.deb"|grep \ 200\ |wc -l
34

EvilOlaf avatar Aug 16 '25 18:08 EvilOlaf