k3c icon indicating copy to clipboard operation
k3c copied to clipboard

Published 20 hours ago • verified publisher icon rancher

exec format error on armhf

Open alexellis opened this issue 5 years ago • 9 comments

Go build is all good, and most of the boot-up, but the CNI drivers bundled are not for the correct architecture. CNI plugins are distributed for armhf/arm64 etc. i.e. https://github.com/alexellis/faasd#hacking-build-from-source

First error:

buildkitd: CNI setup error: fork/exec /var/lib/rancher/k3c/bin/bridge: exec format error

Second error:

FATA[2020-01-14T08:32:38.984245816Z] fork/exec /var/lib/rancher/k3c/bin/unpigz: exec format error

Longer log and system information

pi@k4s-2:~/go/src/github.com/rancher/k3c $ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@k4s-2:~/go/src/github.com/rancher/k3c $ uname -a
Linux k4s-2 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
pi@k4s-2:~/go/src/github.com/rancher/k3c $ sudo ./k3c daemon
INFO[2020-01-14T08:32:37.924746065Z] starting containerd                           revision= version=1.3.2+unknown
INFO[2020-01-14T08:32:38.050408626Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2020-01-14T08:32:38.050658120Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2020-01-14T08:32:38.050759099Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-14T08:32:38.051007612Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-14T08:32:38.057997141Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="modprobe aufs failed: \"modprobe: FATAL: Module aufs not found in directory /lib/modules/4.19.75-v7l+\\n\": exit status 1: skip plugin" type=io.containerd.snapshotter.v1
INFO[2020-01-14T08:32:38.058169545Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2020-01-14T08:32:38.058724717Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/rancher/k3c/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2020-01-14T08:32:38.058801808Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
INFO[2020-01-14T08:32:38.058873659Z] metadata content store policy set             policy=shared
INFO[2020-01-14T08:32:38.059176281Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059251020Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059333815Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2020-01-14T08:32:38.059405832Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059467960Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2020-01-14T08:32:38.059604716Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059666122Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059730991Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059789305Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.059853433Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2020-01-14T08:32:38.060019355Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2020-01-14T08:32:38.060192759Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2020-01-14T08:32:38.061606727Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2020-01-14T08:32:38.061729669Z] loading plugin "io.containerd.grpc.v1.cri"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.062451227Z] Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} UntrustedWorkloadRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} Runtimes:map[runc:{Type:io.containerd.runc.v2 Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false}] NoPivot:false} CniConfig:{NetworkPluginBinDir:/var/lib/rancher/k3c/bin NetworkPluginConfDir:/var/lib/rancher/k3c/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Configs:map[] Auths:map[]} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false} ContainerdRootDir:/var/lib/rancher/k3c/containerd ContainerdEndpoint:/run/k3c/containerd/containerd.sock RootDir:/var/lib/rancher/k3c/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3c/containerd/io.containerd.grpc.v1.cri} 
INFO[2020-01-14T08:32:38.062597872Z] Connect containerd service                   
INFO[2020-01-14T08:32:38.062864199Z] Get image filesystem path "/var/lib/rancher/k3c/containerd/io.containerd.snapshotter.v1.overlayfs" 
INFO[2020-01-14T08:32:38.064326092Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2020-01-14T08:32:38.064468144Z] Start subscribing containerd event           
INFO[2020-01-14T08:32:38.064786823Z] Start recovering state                       
INFO[2020-01-14T08:32:38.064540254Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2020-01-14T08:32:38.065142685Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065206813Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065266330Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065325051Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065382124Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065438289Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065494381Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065549509Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065621082Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065679229Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065735987Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.065792263Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2020-01-14T08:32:38.066515080Z] serving...                                    address=/run/k3c/containerd/containerd.sock.ttrpc
INFO[2020-01-14T08:32:38.066674817Z] serving...                                    address=/run/k3c/containerd/containerd.sock
INFO[2020-01-14T08:32:38.066734798Z] containerd successfully booted in 0.144788s  
INFO[2020-01-14T08:32:38.069722286Z] Start event monitor                          
INFO[2020-01-14T08:32:38.069790859Z] Start snapshots syncer                       
INFO[2020-01-14T08:32:38.069834265Z] Start streaming server                       
INFO[2020-01-14T08:32:38.956955149Z] Bootstrapping data...                        
FATA[2020-01-14T08:32:38.984245816Z] fork/exec /var/lib/rancher/k3c/bin/unpigz: exec format error

alexellis avatar Jan 14 '20 08:01 alexellis

Right now I haven't pushed a multi-arch image for rancher/k3c-data which is pulled on first start.

ibuildthecloud avatar Jan 15 '20 00:01 ibuildthecloud

Sure, I did suspect as much. Let me know when it's good to test.

alexellis avatar Jan 25 '20 14:01 alexellis

@alexellis I get segfaults when I run on armhf (the arm64 builds work for me with the 64 bit kernel however) and so I am not sure if I am just not compiling correctly for that arch but if you could try out https://github.com/rancher/k3c/releases/download/v0.2.0-rc.10/k3c-linux-arm and let me know if it works for you that would be a big help (matching bootstrap image for that arch should be available and working correctly).

dweomer avatar May 14 '20 20:05 dweomer

Hi, sure I can help with that

alexellis avatar May 14 '20 20:05 alexellis 

pi@k4s-1:~ $ chmod +x k3c-linux-arm
pi@k4s-1:~ $ ./k3c-linux-arm 
Illegal instruction
pi@k4s-1:~ $ uname -a
Linux k4s-1 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
pi@k4s-1:~ $ 

pi@k4s-1:~ $ file k3c-linux-arm 
k3c-linux-arm: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=2p3zkA6ahPWTVJowCvb7/N22UmGWq3-gD0tc2tNb6/0aR8ZR7dtI9Ct9bVbQMp/YhJxM7eWPx2wSCVBarzl, BuildID[sha1]=7e8e690cee6d317065b30d8edb577989e277f2ca, stripped

alexellis avatar May 14 '20 20:05 alexellis

For comparison:

pi@k4s-1:~/go/src/github.com/alexellis/output-test $ cat main.go 
package main

func main() {

}
pi@k4s-1:~/go/src/github.com/alexellis/output-test $ /usr/local/go/bin/go build
pi@k4s-1:~/go/src/github.com/alexellis/output-test $ ./output-test 

pi@k4s-1:~/go/src/github.com/alexellis/output-test $ file output-test 
output-test: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=e-qV2CuSr-4GYcG9X7CR/yPP-s_T8bINbPuGpnXWt/Uew6Bu1ulSu9w7KL0nrj/7v5r-fCQin74bALP_MSc, not stripped
pi@k4s-1:~/go/src/github.com/alexellis/output-test $

alexellis avatar May 14 '20 20:05 alexellis 

pi@k4s-1:~ $ chmod +x k3c-linux-arm
pi@k4s-1:~ $ ./k3c-linux-arm 
Illegal instruction
pi@k4s-1:~ $ uname -a
Linux k4s-1 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
pi@k4s-1:~ $ 

pi@k4s-1:~ $ file k3c-linux-arm 
k3c-linux-arm: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=2p3zkA6ahPWTVJowCvb7/N22UmGWq3-gD0tc2tNb6/0aR8ZR7dtI9Ct9bVbQMp/YhJxM7eWPx2wSCVBarzl, BuildID[sha1]=7e8e690cee6d317065b30d8edb577989e277f2ca, stripped

:disappointed: alright, thanks @alexellis ! I'll get it sorted (I hope).

dweomer avatar May 14 '20 21:05 dweomer

Hey folks,

Any update on k3c working on armhf? I'm testing on a Raspberry Pi 4 running Raspberry Pi OS Lite 32-bit

I get the same result as Alex. Illegal instruction

c0dyhi11 avatar Dec 21 '20 20:12 c0dyhi11

I also had trouble starting it on a ARM cpu: segmentation fault. Did try out the ones from k3s: https://github.com/k3s-io/k3s/releases/download/v1.20.2%2Bk3s1/k3s-armhf And that ones worked for me. If you are able to release new version of k3c for armhf, I can test them again. I see definitely a use case for k3c on embedded devices, NAS etc. where only a Docker environment is required. And as Docker is not supporting ARM cpu...

martin-s avatar Jan 17 '21 22:01 martin-s