Unable to install on aarch64, Oracle Linux

[georggrab]

I'm attempting to install on Oracle Linux 8.3, aarch64 architecture.

snapd was installed using epel-release 8, resulting in the following versions:

$ snap version
snap    2.50-1.el8
snapd   2.50-1.el8
series  16
ol      8.3
kernel  5.4.17-2102.201.3.el8uek.aarch64

I receive the following error when installing microk8s 1.21:

$ sudo snap install microk8s --classic --channel=1.21/stable
error: cannot perform the following tasks:
- Run install hook of "microk8s" snap if present (run hook "install":
-----
++ /snap/microk8s/2215/bin/uname -m
+ ARCH=' aarch64'
+ export 'LD_LIBRARY_PATH=/snap/microk8s/2215/lib:/snap/microk8s/2215/usr/lib:/snap/microk8s/2215/lib/ aarch64-linux-gnu:/snap/microk8s/2215/usr/lib/ aarch64-linux-gnu'
+ LD_LIBRARY_PATH='/snap/microk8s/2215/lib:/snap/microk8s/2215/usr/lib:/snap/microk8s/2215/lib/ aarch64-linux-gnu:/snap/microk8s/2215/usr/lib/ aarch64-linux-gnu'
+ export PATH=/snap/microk8s/2215/usr/sbin:/snap/microk8s/2215/usr/bin:/snap/microk8s/2215/sbin:/snap/microk8s/2215/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin:/usr/local/bin
+ PATH=/snap/microk8s/2215/usr/sbin:/snap/microk8s/2215/usr/bin:/snap/microk8s/2215/sbin:/snap/microk8s/2215/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin:/usr/local/bin
+ export OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
+ OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
+ source /snap/microk8s/2215/actions/common/utils.sh
+ cp -r /snap/microk8s/2215/default-args /var/snap/microk8s/2215/args
cp: with --parents, the destination must be a directory
Try 'cp --help' for more information.
-----)

(note the space before "aarch64" in ARCH. Possibly related?)

When I do not specify the channel (I believe this currently installs 1.20?), I get the following error (same for 1.19, 1.18)

$ sudo snap install --classic microk8s
error: cannot perform the following tasks:
- Run install hook of "microk8s" snap if present (run hook "install":
-----
++ /snap/microk8s/2217/bin/uname -m
+ ARCH=
-----)

[ktsakalozos]

Do other snaps work properly?

Errors in cp -r /snap/microk8s/2215/default-args /var/snap/microk8s/2215/args or uname are very strange. As if the snap did not deploy on the right path or not interacting properly with the host system.

[georggrab]

I'm running another Snap on this node (nebula), this one works just fine.

[gespispace]

I also have this problem. I try start in oracle cloud on new arm Instances.

[NikolaiNikolovskiNik]

I also have the same issue. Tried on OL8.4 raspberypi image

sudo snap install microk8s --classic
error: cannot perform the following tasks:
- Run install hook of "microk8s" snap if present (run hook "install":
-----
++ /snap/microk8s/2347/bin/uname -m
+ ARCH=' aarch64'
+ export 'LD_LIBRARY_PATH=/snap/microk8s/2347/lib:/snap/microk8s/2347/usr/lib:/snap/microk8s/2347/lib/ aarch64-linux-gnu:/snap/microk8s/2347/usr/lib/ aarch64-linux-gnu'
+ LD_LIBRARY_PATH='/snap/microk8s/2347/lib:/snap/microk8s/2347/usr/lib:/snap/microk8s/2347/lib/ aarch64-linux-gnu:/snap/microk8s/2347/usr/lib/ aarch64-linux-gnu'
+ export PATH=/snap/microk8s/2347/usr/sbin:/snap/microk8s/2347/usr/bin:/snap/microk8s/2347/sbin:/snap/microk8s/2347/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin:/usr/local/bin
+ PATH=/snap/microk8s/2347/usr/sbin:/snap/microk8s/2347/usr/bin:/snap/microk8s/2347/sbin:/snap/microk8s/2347/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin:/usr/local/bin
+ export OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
+ OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
+ source /snap/microk8s/2347/actions/common/utils.sh
+ cp -r /snap/microk8s/2347/default-args /var/snap/microk8s/2347/args
cp: with --parents, the destination must be a directory
Try 'cp --help' for more information.
-----)

[maxsatula]

Here are my findings: Microk8s's snap squashfs image contains coreutils version 8.28, which provides uname and cp utilities cp -r /snap/microk8s/2347/default-args /var/snap/microk8s/2347/args command is expected to create args directory in 2347, but apparently cp version 8.28 cannot do that. I tested with a cp version installed on the host, which was 8.30, and it would work well. Also, uname version 8.28 prints an extra space in front of aarch64, while version 8.30 does not. I guess first thing to try is to build a new microk8s snap image with latest version of coreutils.

So, to test that, I took a snap image from cache, unpacked it, replaced cp and uname with binaries on the host, and packed a new snap image. Now it fails with Segmentation fault trying to run openssl, well, had to refresh host's binary and libssl and libcrypto libraries into the image too. Then it segfaulted on sed, replaced it as well. Ugh! NOW, apparently it installed (finally), but service fails to start with "Failed to start Service for snap application microk8s.daemon-containerd." I need a break. I will keep you updated once I have more time to keep troubleshooting

[maxsatula]

Also had to replace /bin/sleep with the one on the host, as it segfaulted too... Now I've seen the services running for a while, then it failed with

Start snap "microk8s" (unset) services (systemctl command [start snap.microk8s.daemon-containerd.service] failed with exit status 1: Job for snap.microk8s.daemon-containerd.service failed because a timeout was exceeded.

[maxsatula]

Even after replacing more failing binaries/libs and finally installing microk8s a simple microk8s status segfaults on calling python3. I'm giving up. That seems to be a general snap problem, not microk8s specific. Snapcraft segfaults as well. As someone wrote on the internet, paraphrased as I lost the original, "snap is Ubuntu oriented, and it is not linux distribution agnostic as snap folks want to believe"

Also, looks pretty much related issue: https://forum.snapcraft.io/t/classic-snaps-failing-on-ubuntu-17-10/2324/14

[joedborg]

Thanks for the investigation @maxsatula. I've reproduced all you've seen as well. I believe this to be a bug in snapd (or maybe snapcraft) and have opened a post on their forum https://forum.snapcraft.io/t/different-behaviour-between-ubuntu-and-rhel-based-on-arm/27598

I'm not convinced it's the exact same as the thread you've linked as this snap works fine on x86-64 across Ubuntu and RHEL based OSes, but it's certainly very similar. Let's see!

Thanks again, your participation is much appreciated.

[hasan4791]

On the other hand, I'm getting this error with 1.23 stable release.

- Run install hook of "microk8s" snap if present (run hook "install":
-----
++ /snap/microk8s/2954/bin/uname -m
/var/lib/snapd/snap/microk8s/2954/meta/hooks/install: line 5: /snap/microk8s/2954/bin/uname: cannot execute binary file: Exec format error
+ ARCH=
-----)

I tried running uname command from the following directory which works fine.

[root@instance-20220130-1745 temp]# ls -l /snap/core18/current/bin/uname
-rwxr-xr-x. 1 root root 30872 Jan 18  2018 /snap/core18/current/bin/uname
[root@instance-20220130-1745 temp]# /snap/core18/current/bin/uname -m
aarch64

Next, I tried to copy the files from the mounted directory of microk8s to confirm about the binary status and there I was getting similar error

[root@instance-20220130-1745 temp]# ls -l 2954/bin/uname
-rwxr-xr-x. 1 root root 34280 Jan 30 13:50 2954/bin/uname
[root@instance-20220130-1745 temp]# 2954/bin/uname -m
-bash: 2954/bin/uname: cannot execute binary file: Exec format error
[root@instance-20220130-1745 temp]#

I had a doubt whether this particular binary is having issue and so I tried to execute the list of all common commands and all got failed

[root@instance-20220130-1745 temp]# 2954/bin/pwd
-bash: 2954/bin/pwd: cannot execute binary file: Exec format error
[root@instance-20220130-1745 temp]# 2954/bin/mktemp
-bash: 2954/bin/mktemp: cannot execute binary file: Exec format error
[root@instance-20220130-1745 temp]# 2954/bin/ls
-bash: 2954/bin/ls: cannot execute binary file: Exec format error
[root@instance-20220130-1745 temp]# 2954/bin/mkdir
-bash: 2954/bin/mkdir: cannot execute binary file: Exec format error
[root@instance-20220130-1745 temp]# 2954/bin/cp
-bash: 2954/bin/cp: cannot execute binary file: Exec format error
[root@instance-20220130-1745 temp]# 2954/bin/mv
-bash: 2954/bin/mv: cannot execute binary file: Exec format error

After checking the supplied binary for the target system, it looks fine

[root@instance-20220130-1745 temp]# file 2954/bin/uname
2954/bin/uname: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /snap/core18/current/lib/ld-linux-aarch64.so.1, BuildID[sha1]=0ac693a7aac54df14acd1e6667c07d9809eda1b2, for GNU/Linux 3.7.0, stripped
[root@instance-20220130-1745 temp]#

Is there some issue on the microk8s side or am i missing anything here?

Commands used for installation of snap & microk8s

    1  dnf update
    2  yum-config-manager --enable ol8_developer_EPEL
    3  dnf update
    4  dnf install snapd
    5  systemctl enable --now snapd.socket
    6  ln -s /var/lib/snapd/snap /snap
    7  snap install microk8s --classic

[masterrob94]

Update: microk8s works on Red Hat Compatible Kernel (RHCK) kernel-4.18.0-348, but not on Unbreakable Enterprise Kernel (uek) kernel-uek-5.4.17-2136.300.7 on Oracle Linux 8 with Update 5

https://blogs.oracle.com/linux/post/announcing-the-release-of-oracle-linux-8-update-5

You can change this by editing the boot config to use kernel 4.18

[maxsatula]

You can change this by editing the boot config to use kernel 4.18

Thanks for that, better than nothing, however that is more of a workaround rather than a solution. I would prefer to stay with kernel-uek and use microk8s at the same time :(

[hyacin75]

Update: microk8s works on Red Hat Compatible Kernel (RHCK) kernel-4.18.0-348, but not on Unbreakable Enterprise Kernel (uek) kernel-uek-5.4.17-2136.300.7 on Oracle Linux 8 with Update 5

https://blogs.oracle.com/linux/post/announcing-the-release-of-oracle-linux-8-update-5

You can change this by editing the boot config to use kernel 4.18

Sorry, not the right place to ask I know, but Google is leaving me at a loss. How the hell do you install the RHCK?! I've searched high and low, jumped through some crazy hoops to get instances up to install it on, and even switched to Oracle Linux developer's version or whatever it is called ... the AMD64 version has RHCK in /boot at install, but it is nowhere to be found on the ARM version, which is obviously what we're discussing here as we're talking about problems installing snap on ARM! :-/

Any help would be GREATLY appreciated!

I'm about to install the source and just build it from there, but I'm feeling like that's a really bad idea that is going to leave me with an unbootable VM I need to just blow up and start fresh from.

[tgie]

Thanks for that, better than nothing, however that is more of a workaround rather than a solution. I would prefer to stay with kernel-uek and use microk8s at the same time :(

It´s fixed: https://github.com/oracle/linux-uek/issues/13

[maxsatula]

@tgie that seems to fix for x86_64, aarch64 still has a problem

[xin3liang]

Any update? I have the same problem on CentOS 8.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.