microshift icon indicating copy to clipboard operation
microshift copied to clipboard
verified publisher icon openshift

[BUG] Observed a panic: runtime error: invalid memory address or nil pointer dereference

Open rbaumgar opened this issue 3 years ago • 6 comments

What happened:

microshift panic error during startup in loop.

What you expected to happen:

microshift get ready

How to reproduce it (as minimally and precisely as possible):

  1. installed Fedora IOT 36 on raspberry 3B. Fedora Linux 36.20220731.0 (IoT Edition)
  2. install cri-o 1.21, microshift-4.8.0-2022_04_20_141053.fc36.aarch64
  3. systemctl enable crio microshift

Anything else we need to know?:

Environment:

  • Microshift version (use microshift version): MicroShift Version: 4.8.0-0.microshift-2022-04-20-141053 Base OKD Version: 4.8.0-0.okd-2021-10-10-030117

  • Hardware configuration: Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: ARM BIOS Vendor ID: Unknown Model name: Cortex-A53 BIOS Model name: Unknown CPU @ 0.0GHz BIOS CPU family: 2 Model: 4 Thread(s) per core: 1 Core(s) per cluster: 4 Socket(s): 1 Cluster(s): 1 Stepping: r0p4 CPU(s) scaling MHz: 68% CPU max MHz: 1400.0000 CPU min MHz: 600.0000 BogoMIPS: 38.40 Flags: fp asimd evtstrm crc32 cpuid Caches (sum of all):
    L1d: 128 KiB (4 instances) L1i: 128 KiB (4 instances) L2: 512 KiB (1 instance) NUMA:
    NUMA node(s): 1 NUMA node0 CPU(s): 0-3 Vulnerabilities:
    Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Spec store bypass: Not affected Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Not affected Srbds: Not affected Tsx async abort: Not affected [root@fedora ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS mmcblk0 179:0 0 28.9G 0 disk ├─mmcblk0p1 179:1 0 501M 0 part /boot/efi ├─mmcblk0p2 179:2 0 1G 0 part /boot └─mmcblk0p3 179:3 0 27.4G 0 part /var /usr / /sysroot zram0 252:0 0 925M 0 disk [SWAP]

  • OS (e.g: cat /etc/os-release): NAME="Fedora Linux" VERSION="36.20220731.0 (IoT Edition)" ID=fedora VERSION_ID=36 VERSION_CODENAME="" PLATFORM_ID="platform:f36" PRETTY_NAME="Fedora Linux 36.20220731.0 (IoT Edition)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:36" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/" SUPPORT_URL="https://ask.fedoraproject.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=36 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=36 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="IoT Edition" VARIANT_ID=iot OSTREE_VERSION='36.20220731.0'

  • Kernel (e.g. uname -a): Linux microshift 5.18.13-200.fc36.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Jul 22 13:44:25 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Relevant Logs

[microshift.log] (https://github.com/openshift/microshift/files/9232500/microshift.log)

rbaumgar avatar Aug 01 '22 09:08 rbaumgar

The microshift binary from Apr 20, 2022 installed in /usr/bin by the rpm installer does not work with Fedora 36. It causes a crash-loop as seen with journalctl command. Replacing the microshift binary installed by the rpm with a newer binary from May 11, 2022 downloaded from https://github.com/openshift/microshift/releases/ works. Note that the microshift-linux-arm64 binaries from 05-11-2022 Latest Nightly Build or 04-20-2022 (both 4.8.0-0.microshift-2022-04-20-182108 and 4.8.0-0.microshift-2022-04-20-141053) work. The microshift version installed by rpm shows 4.8.0-0.microshift-2022-04-20-141053 but it does not work. I built the microshift binary from sources and that works too.

I also mentioned this in my blog https://community.ibm.com/community/user/cloud/blogs/alexei-karve/2022/06/20/microshift-21 search for "Replace MicroShift binary"

thinkahead avatar Aug 01 '22 10:08 thinkahead

I used this a workaround: (problem Fedora IOT is only read-only FS) curl -L https://github.com/openshift/microshift/releases/download/nightly/microshift-linux-arm64 > /usr/local/bin/microshift chmod +x /usr/local/bin/microshift /usr/local/bin/microshift version sed -i "s|/usr/bin|/usr/local/bin|" /etc/systemd/system/microshift.service systemctl daemon-reload

rbaumgar avatar Aug 01 '22 12:08 rbaumgar

Now I receive another Fatalf.. microshift.log .

rbaumgar avatar Aug 01 '22 14:08 rbaumgar

I had used cri-o:1.23

rpm-ostree ex module enable cri-o:1.23 # Experimental enable
rpm-ostree install cri-o cri-tools microshift

thinkahead avatar Aug 01 '22 15:08 thinkahead

Now I have a raspberry 4 with 4GB. Installed Fedora IOT 36 Installed cri-o 1.23 Microshift nightly build --> working!

oc get node -o wide
NAME         STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                   KERNEL-VERSION            CONTAINER-RUNTIME
microshift   Ready    <none>   4m13s   v1.21.0   192.168.1.4   <none>        Fedora Linux 36.20220618.0 (IoT Edition)   5.18.5-200.fc36.aarch64   cri-o://1.23.0

rbaumgar avatar Aug 02 '22 15:08 rbaumgar

Right, use the nightly build for now.

Fedora36 provided the wrong golang compiler version that seems to make some of the components crash. This was fixed on the nightly build by enforcing an specific golang version.

We are working on the product MicroShift, afterwards you should see activity again on the community version.

mangelajo avatar Aug 08 '22 14:08 mangelajo

We're focusing on RHEL support for the moment and the main branch has moved on quite a bit from the 4.8 release branch. You could try compiling the main branch, but it may be a few months before the team returns to focusing on Fedora support.

dhellmann avatar Aug 21 '22 16:08 dhellmann