firmware icon indicating copy to clipboard operation
firmware copied to clipboard

After updating 32 bit Raspi OS Pi 4 is on 64bit kernel

Open edegroot-nl opened this issue 2 years ago • 50 comments

After updating Raspberry PI OS to the last kernel update, the kernel is switched to 64 bit.

$ uname -a Linux 5.15.84-v7l+ #1613 SMP Thu Jan 5 12:01:26 GMT 2023 armv7l GNU/Linux

$ sudo apt update .... $ sudo apt upgrade

The following packages will be upgraded: curl libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libcamera-apps-lite libcamera0 libcurl3-gnutls libcurl4 libnss3 libtiff5 linux-libc-dev raspberrypi-bootloader raspberrypi-kernel raspi-config vcdbg 16 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 108 MB/111 MB of archives. After this operation, 3,879 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://archive.raspberrypi.org/debian bullseye/main armhf raspberrypi-kernel armhf 1:1.20230317-1 [102 MB] Get:2 http://archive.raspberrypi.org/debian bullseye/main armhf linux-libc-dev armhf 1:1.20230317-1 [1,119 kB] Get:3 http://archive.raspberrypi.org/debian bullseye/main armhf raspberrypi-bootloader armhf 1:1.20230317-1 [4,542 kB] Get:4 http://archive.raspberrypi.org/debian bullseye/main armhf vcdbg armhf 1:1.20230317-1 [213 kB] ...... Setting up raspberrypi-kernel (1:1.20230317-1) .. .....

$ sudo reboot .... $ uname -a Linux 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux

For me this is a problem while running some software that needs 32bit.

arm_64bit=0 added tot /boot/config.txt and everything fine now. Why not respect the 32bit kernel when already active and change the setting to arm_64bit=1 if needed?

edegroot-nl avatar Mar 18 '23 15:03 edegroot-nl

The switch to running a 64-bit kernel by default on Pi 4 is intentional. We believe it gives a better experience with few drawbacks. As you've found, you can revert to the 32-bit kernel by adding arm_64bit=0 to config.txt.

See https://forums.raspberrypi.com/viewtopic.php?p=2088935#p2088935 for more details.

Which software needs to be run on a 32-bit kernel?

pelwell avatar Mar 18 '23 15:03 pelwell

Which software needs to be run on a 32-bit kernel?

Unifi Controller with MongoDB and Keepalived for example.

edegroot-nl avatar Mar 18 '23 16:03 edegroot-nl

What happens when you try to run them? Do you know the technical limitations?

pelwell avatar Mar 18 '23 16:03 pelwell

What happens when you try to run them? Do you know the technical limitations?

Hi Phil, For MongoDB quite old version but needed by unifi:

It gets an bus error. I don't know the limitations.

[2023-03-18T14:54:20,673] INFO db - DbServer stopped [2023-03-18T14:54:24,850] INFO db - Sat Mar 18 14:54:24.849 [2023-03-18T14:54:24,853] INFO db - Sat Mar 18 14:54:24.850 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. [2023-03-18T14:54:24,854] INFO db - Sat Mar 18 14:54:24.850 [2023-03-18T14:54:24,854] INFO db - Sat Mar 18 14:54:24.854 Invalid access at address: 0x1c076ee from thread: [2023-03-18T14:54:24,855] INFO db - [2023-03-18T14:54:24,855] INFO db - Sat Mar 18 14:54:24.854 Got signal: 7 (Bus error).

Keepalived is the default version from the repo:

Mar 18 12:29:15 edg-rpi03 Keepalived[4765]: pid 13817 exited due to signal 7 (Bus error) Mar 18 12:29:15 edg-rpi03 Keepalived[4765]: VRRP child process(13817) died: Respawning Mar 18 12:29:15 edg-rpi03 Keepalived[4765]: Restart of VRRP process delayed 60 seconds to limit respawn rate

II'm fine with this solution to go back to 32 bit right now.

edegroot-nl avatar Mar 18 '23 16:03 edegroot-nl

@edegroot-nl You must be using an old version of UniFi Controller!

You can use current versions of UniFi Controller (7.3+ at least) with MongoDB 3.6+, which is available for aarch64. See this project for more info: https://github.com/ryansch/docker-unifi-rpi

soren121 avatar Mar 20 '23 18:03 soren121

@soren121

I'm using UniFi Network Application 7.3.83 stable released January 30th 2023, which is still supported with an older MongoDB version. The default version from the 32-bit repo.

I will install the standard 64-bit version of RaspBerry PI OS later if everything runs without issues with aarch64 packages.

edegroot-nl avatar Mar 20 '23 19:03 edegroot-nl

The real problem is that people running production systems rely on updates to fix security problems, making unannounced changes of this kind which break things a real problem. Fortunately, there is a fallback, but it would have been better to let people know about the change and tell them how to stay on a 32 bit kernel.

ralphrmartin avatar Mar 22 '23 06:03 ralphrmartin

I am also not exited about this change. Espressif esp32 tools are only available for 32bit platforms and a usual apt upgrade just renders the setup unusable, unless this is fixed as mentioned above.

VirusPilot avatar Mar 24 '23 17:03 VirusPilot

It what way are the esp32 tools dependent on a 32-bit kernel? Very few 32-bit apps should care what mode the kernel is running in.

pelwell avatar Mar 24 '23 17:03 pelwell

After putting arm_64bit=0 into config.txt the whole system is unable to boot. I removed the statement and the Pi boots fine but PIVCCU3 is not able to work because of missing kernel modules which are not (and most likely will never be) compatible with a 64bit kernel. Any suggestions why the raspberry doesn't boot anymore? The kernel modules were automatically removed after apt-get upgrade.

Tylli avatar Mar 24 '23 17:03 Tylli

It what way are the esp32 tools dependent on a 32-bit kernel? Very few 32-bit apps should care what mode the kernel is running in.

The esp32 tools are simply parsing $arch and if the result is aarch64, they won't work.

VirusPilot avatar Mar 24 '23 17:03 VirusPilot

PortAudio isn't working properly after this update. With adding arm_64bit=0 into config.txt our application (using portaudio) is working but without it will crash. The problem we have ist that port audio will report different count of devices with Pa_GetDeviceCount(). In our logs we see after calling Pa_Initialize() several time this failure: Expression 'GetExactSampleRate( hwParams, &defaultSr )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 895 So it seem this change has more side effects than expected.

STMAndreas avatar Mar 29 '23 15:03 STMAndreas

Another example: GlusterFS fails with SIGBUS - it took me three nights to figure out what was wrong - luckily I noticed something was wrong before updating all nodes. Definitely not a great idea to push this update without a "breaking changes" notice.

fabiopuddu avatar Apr 05 '23 23:04 fabiopuddu

Doesn't this change mean that uname -m is returning the wrong value? libc etc are still 32 bit so any build that attempts to detect the architecture using uname will fail, right?

AndyA avatar Apr 06 '23 16:04 AndyA

OK, it's returning the correct value for the kernel - but that doesn't seem super helpful in userspace.

AndyA avatar Apr 06 '23 17:04 AndyA

@edegroot-nl thanks a lot for posting the keepalived issue on here. I would've never figured out what broke it. Can anyone point me into the right direction on how to get it back running when I wanna stick with the 64-bit kernel?

TheHoolio avatar Apr 09 '23 10:04 TheHoolio

Doesn't this change mean that uname -m is returning the wrong value? libc etc are still 32 bit so any build that attempts to detect the architecture using uname will fail, right?

uname asks about the kernel. It's the wrong tool for this job.

getconf LONG_BIT

would be a better way to query if userland is 32 or 64 bit.

popcornmix avatar Apr 09 '23 10:04 popcornmix

@pelwell does the 32-bit OS contains all the userspace tools to build out-of-tree kernel modules in 64-bit?

HinTak avatar Apr 13 '23 18:04 HinTak

You should just need sudo apt-get install gcc-aarch64-linux-gnu, but I don't build out-of-tree modules so there might be something else to install.

pelwell avatar Apr 13 '23 19:04 pelwell

@pelwell thanks. There are also hints that matching kernel header package isn't available via apt-get. I see it on archive.raspberrypi.org, but is there an oversight somewhere on this - ie the arm64 kernel headers package isn't accessible via apt-get?

HinTak avatar Apr 13 '23 23:04 HinTak

@XECDesign are arm64 kernel headers available on 32-bit RPiOS through apt?

popcornmix avatar Apr 14 '23 10:04 popcornmix

@XECDesign are arm64 kernel headers available on 32-bit RPiOS through apt?

Unfortunately not yet. https://github.com/raspberrypi/linux/issues/5408

XECDesign avatar Apr 17 '23 10:04 XECDesign

Until that package is available it may be possible to use rpi-source

popcornmix avatar Apr 17 '23 11:04 popcornmix

This change breaks installation scripts that rely on uname -m to decide whether to install 32-bit or 64-bit software. What's the supported way to determine whether you're running the 64-bit kernel on a 32-bit OS vs a 64-bit OS?

ebaauw avatar Apr 26 '23 17:04 ebaauw

@ebaauw I think one of apt-* or deb-* family has a --archtecture query option. It still surprises me somewhat, how a 64-bit kernel get passed that, but the header package does not.

HinTak avatar Apr 26 '23 17:04 HinTak

What's the supported way to determine whether you're running the 64-bit kernel on a 32-bit OS vs a 64-bit OS?

Already answered in this thread.

pelwell avatar Apr 26 '23 17:04 pelwell

Sorry, missed that. Thanks!

ebaauw avatar Apr 26 '23 18:04 ebaauw

homebridge fails as well

oilcan-productions avatar Apr 26 '23 18:04 oilcan-productions

@oilcan-productions "fails" doesn't help much.

Describe exactly what you are doing, what should happen, what currently happens, and if setting arm_64bit=0 avoids the issue.

popcornmix avatar Apr 26 '23 19:04 popcornmix

@oilcan-productions "fails" doesn't help much.

Describe exactly what you are doing, what should happen, what currently happens, and if setting arm_64bit=0 avoids the issue.

https://github.com/homebridge/homebridge/issues/3349

and yes it fixes the issue for homebridge. I do agree with the sentiment of others that a breaking change like that should not happen

oilcan-productions avatar Apr 26 '23 19:04 oilcan-productions