boot2podman
boot2podman copied to clipboard
boot2podman
Package for Fedora/ARM
Hello @afbjorklund , I would very much like to run k3s/cri-o/podman on ARM
- Raspberry PI at home and
- AWS Graviton at work I can also try to get it into the fedora repo (I help out there sometimes)
Have you already considered this? Do you have suggestions? Thanks!!
It should be possible. There is a version of tinycorelinux for the Raspberry Pi (called piCore):
http://www.tinycorelinux.net/9.x/armv6/releases/RPi/
So basically it needs the same kind of custom kernel, and all the packages need rebuilding...
Luckily one doesn't have to do it on the device: https://github.com/afbjorklund/raspbian-proot
Sounds like a worthy Hacktober project, and I do have the hardware for it. Here is the cluster:
Currently they are running Hypriot, but the regular k8s installation is getting too big for them.
However, if you just want something to run on the device today then you should look at k3os! It uses containerd rather than cri-o, and doesn't include docker by default (you could add it)
Or you could install Fedora ARM, and then install podman and cri-o from the RPM packages. I don't think k3s is packaged yet, so you would have to use the regular k3s binary (for armhf)
More details for Fedora: https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi
-
podman: https://koji.fedoraproject.org/koji/packageinfo?packageID=26289
-
cri-o: https://koji.fedoraproject.org/koji/packageinfo?packageID=24240
I think that cri-o might actually be missing from the later Fedora versions, because modules.
-
k3s (32-bit): https://github.com/rancher/k3s/releases/latest/download/k3s-armhf
-
k3s (64-bit): https://github.com/rancher/k3s/releases/latest/download/k3s-arm64
To start it up: k3s server --container-runtime-endpoint /var/run/crio/crio.sock
Yes, cri-o is missing from fc30 for ARM....amazing. Thanks for the other links and the context, this is all very encouraging. I have 2 spare Pi 3+ at the moment.... and quite a few ARM instances in EC2. I like the idea of podman because I may have strangers running builds on my instances and I like the idea of a docker runtime without root privileges.
Please let me know if there is anything I can do to help you.
Cheers, Blaise
On Wed, Oct 2, 2019 at 1:47 PM Anders Björklund [email protected] wrote:
More details for Fedora: https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi
podman: https://koji.fedoraproject.org/koji/packageinfo?packageID=26289
cri-o: https://koji.fedoraproject.org/koji/packageinfo?packageID=24240
I think that cri-o might actually be missing from the later Fedora versions, because modules.
https://github.com/rancher/k3s/releases/download/v0.9.1/k3s-armhf
https://github.com/rancher/k3s/releases/download/v0.9.1/k3s-arm64
To start it up: k3s server --container-runtime-endpoint /var/run/crio/crio.sock
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/boot2podman/boot2podman/issues/18?email_source=notifications&email_token=ALSRZWIDP6FCLEAEI6HOCTLQMUCF3A5CNFSM4I42KWRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAGD7EI#issuecomment-537673617, or mute the thread https://github.com/notifications/unsubscribe-auth/ALSRZWJQ6H5SPP7EJS266STQMUCF3ANCNFSM4I42KWRA .
--
Blaise Pabon
Sales Engineer, Enterprise
O(650) 503-6743 | M (650) 450-4220
https://www.linkedin.com/in/blaisepabon https://twitter.com/sumologic https://www.glassdoor.com/Overview/Working-at-Sumo-Logic-EI_IE621848.11,21.htm https://www.sumologic.com/ https://www.sumologic.com/illuminate/
There is a Fedora ISO for Boot2Podman available: https://boot2podman.github.io/2018/12/07/fedora-iso-variant.html
It might be quicker to do an ARM image based on that kickstart: https://github.com/boot2podman/boot2podman-fedora-iso/blob/master/fedora.template
Need to make an updated Fedora 30 ISO build anyway, so could take a look at what is needed for an armv7. Still more fun to build one from scratch, but... Most likely this ISO variant would use the official kubeadm rather than k3s, though. Might change that for ARM.
Hi Anders,
What a great idea! I did not know about the kickstart, I like that a lot.
I don't have a problem with the kubeadm, I was using k3s for its
simplicity.
The AWS arm64 images are cheap, so I can be generous with them. My immediate objective is to be able to create realistic logging data, so that I can make simulations and demos. (I work for a company that does log aggregation). My boss is a big fan of opensource projects, so I run a private gitlab server and I can give you access if you want to experiment with builds, benchmarks or whatever.
For example, I recently considered setting up a MQTT broker in case that could be useful to people without highly available internet access. or maybe help out with https://aprs.fi/#!lat=37.0156&lng=-121.5779 Do things like this interest you?
Cheers, Blaise
On Wed, Oct 2, 2019 at 11:09 PM Anders Björklund [email protected] wrote:
There is a Fedora ISO for Boot2Podman available: https://boot2podman.github.io/2018/12/07/fedora-iso-variant.html
It might be quicker to do an ARM image based on that kickstart:
https://github.com/boot2podman/boot2podman-fedora-iso/blob/master/fedora.template
Need to make an updated Fedora 30 ISO build anyway, so could take a look at what is needed for an armv7. Still more fun to build one from scratch, but... Most likely this ISO variant would use the official kubeadm rather than k3s, though. Might change that for ARM.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/boot2podman/boot2podman/issues/18?email_source=notifications&email_token=ALSRZWLDCWCFTHZOWNYATXDQMWEBTA5CNFSM4I42KWRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAHDNGY#issuecomment-537802395, or mute the thread https://github.com/notifications/unsubscribe-auth/ALSRZWKBDW5QHRJG7U4VAGDQMWEBTANCNFSM4I42KWRA .
--
Blaise Pabon
Sales Engineer, Enterprise
O(650) 503-6743 | M (650) 450-4220
https://www.linkedin.com/in/blaisepabon https://twitter.com/sumologic https://www.glassdoor.com/Overview/Working-at-Sumo-Logic-EI_IE621848.11,21.htm https://www.sumologic.com/ https://www.sumologic.com/illuminate/
I don't have a problem with the
kubeadm, I was using k3s for its simplicity.
The main reason for going with k3s for this arm image would be the footprint.
While k8s (kubeadm) requires 2G of memory, k3s can get away with 512M....
Another adaptation seems to be that ARM images don't use tmpfs for /tmp:
# Because memory is scarce resource in most arm systems we are differing from the Fedora
# default of having /tmp on tmpfs.
echo "Disabling tmpfs for /tmp."
systemctl mask tmp.mount
-
https://pagure.io/fedora-kickstarts/blob/f30/f/fedora-disk-base.ks
-
https://pagure.io/fedora-kickstarts/blob/f30/f/fedora-arm-base.ks
Hi @afbjorklund , so I also hang out in the OpenFAAS group and they were just talking about https://rancher.com/blog/2019/announcing-k3os-kubernetes-operating-system/ and how it might be cool to see it run on Rpi...
Tested with the previous piCore, and it boots and runs fine on the Raspberry Pi 3 (armv7).
( '>')
/) TC (\ Core is distributed with ABSOLUTELY NO WARRANTY.
(/-_--_-\) www.tinycorelinux.net
tc@box:~$ uname -a
Linux box 4.9.22-piCore-v7 #1 SMP Sat Apr 15 12:27:07 UTC 2017 armv7l GNU/Linux
tc@box:~$ free -m
total used free shared buffers cached
Mem: 923 156 767 9 8 81
-/+ buffers/cache: 66 856
Swap: 222 0 222
No issues with loading the "compiletc" package either, so at least the basics are working:
gcc (piCore) 7.1.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Next up is compiling a new kernel (with container support), and building go for arm...
Lots of good Linux config hints to be found in the Hypriot kernel: https://blog.hypriot.com/post/verify-kernel-container-compatibility/
tc@box:~$ ./check-config.sh
info: reading kernel config from /proc/config.gz ...
Generally Necessary:
- cgroup hierarchy: nonexistent??
(see https://github.com/tianon/cgroupfs-mount)
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_MEMCG_SWAP: missing
- CONFIG_MEMCG_SWAP_ENABLED: missing
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
- "overlay":
- CONFIG_VXLAN: enabled (as module)
- CONFIG_BRIDGE_VLAN_FILTERING: missing
Optional (for encrypted networks):
- CONFIG_CRYPTO: enabled
- CONFIG_CRYPTO_AEAD: enabled
- CONFIG_CRYPTO_GCM: enabled (as module)
- CONFIG_CRYPTO_SEQIV: enabled
- CONFIG_CRYPTO_GHASH: enabled (as module)
- CONFIG_XFRM: enabled
- CONFIG_XFRM_USER: enabled
- CONFIG_XFRM_ALGO: enabled
- CONFIG_INET_ESP: enabled (as module)
- CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
- "ipvlan":
- CONFIG_IPVLAN: missing
- "macvlan":
- CONFIG_MACVLAN: enabled (as module)
- CONFIG_DUMMY: enabled (as module)
- "ftp,tftp client in container":
- CONFIG_NF_NAT_FTP: enabled (as module)
- CONFIG_NF_CONNTRACK_FTP: enabled (as module)
- CONFIG_NF_NAT_TFTP: enabled (as module)
- CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
- "aufs":
- CONFIG_AUFS_FS: missing
- "btrfs":
- CONFIG_BTRFS_FS: enabled (as module)
- CONFIG_BTRFS_FS_POSIX_ACL: enabled
- "devicemapper":
- CONFIG_BLK_DEV_DM: enabled (as module)
- CONFIG_DM_THIN_PROVISIONING: enabled (as module)
- "overlay":
- CONFIG_OVERLAY_FS: enabled (as module)
- "zfs":
- /dev/zfs: missing
- zfs command: missing
- zpool command: missing
Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
http://www.tinycorelinux.net/9.x/armv6/releases/RPi/src/kernel/
We don't have to build Go, but I prefer building from source...
However the official Google binary (for armv6l) works just fine:
tc@box:~$ go version
go version go1.12.10 linux/arm
https://dl.google.com/go/go1.12.10.linux-armv6l.tar.gz
Well, it is still only the older version (9.0) - but at least the container version now works too:
sudo podman run -it boot2podman-docker-tinycore.bintray.io/tinycore:9.0-armv6
See: https://github.com/boot2podman/boot2podman/tree/master/containers/tinycore-arm
I think I might have left the kernel modules in there, which is why it is twice as big (as x86_64)
Unfortunately Go programs, including the Go compiler, are not entirely stable under qemu-arm
note that qemu is not reliable when running go programs. qemu's emulation is too glibc specific.
" I think we lost the ability to run Go binaries with qemu-arm long time ago." (said in 2015)
So probably have to cross-compile the go compiler for armv6, using the regular x86_64 image.
Building natively on the Raspberry Pi has its own issues, but now the go.tcz is ready at least.
https://dl.bintray.com/boot2podman/tinycorelinux/9.x/armv6/tcz/go.tcz
Most of the packages have been recompiled, but want a cross-compiler in order to build kernel.
$ /usr/local/bin/podman version
Version: 1.6.2-dev
RemoteAPI Version: 1
Go Version: go1.12.10
OS/Arch: linux/arm
$ /usr/local/lib/podman/conmon --version
conmon version 2.0.3-dev
commit: bc758d8bd98a29ac3aa4f62a886575bfec0e39a1
$ /usr/local/bin/runc --version
runc version 1.0.0-rc9+dev
commit: c1485a1e88f853e9c2cd3d51eac6d410fed24df4
spec: 1.0.1-dev
Building on the rpi takes a long time, so better to use a arm-linux-gnueabihf cross-compiler ?
Also some of the TCZ packages available in Core 9.x are missing from piCore 9.x, unfortunately...
- compile_libseccomp
- compile_libassuan
- compile_gpgme
- compile_meson
Upgrading to Tiny Core Linux 10.x and building for aarch64 (arm64) will be saved for another time.
@blaise-sumo : the fedora packaging for golang seems weird, the cri-o package is a big mess...
So I just did a small local rpm of /usr/bin/k3s, just to have it available as a (NoSource) package.
Name: k3s
Version: 0.9.1
Release: 0%{?dist}
Summary: Lightweight Kubernetes
Group: Applications/Internet
License: ASL 2.0
URL: https://k3s.io
%ifarch x86_64
Source: https://github.com/rancher/k3s/releases/download/v%{version}/%{name}
%endif
%ifarch armv7hl
Source: https://github.com/rancher/k3s/releases/download/v%{version}/%{name}-armhf
%endif
%ifarch aarch64
Source: https://github.com/rancher/k3s/releases/download/v%{version}/%{name}-arm64
%endif
NoSource: 0
Source1: k3s.service
Source2: k3s.service.env
%description
"k3s - 5 less than k8s"
Easy to install, half the memory, all in a binary less than 40mb.
%install
install -Dp -m 755 %{SOURCE0} %{buildroot}%{_bindir}/%{name}
install -Dp -m 644 %{SOURCE1} %{buildroot}/etc/systemd/system/k3s.service
install -Dp -m 400 %{SOURCE2} %{buildroot}/etc/systemd/system/k3s.service.env
%files
%{_bindir}/%{name}
/etc/systemd/system/k3s.service
%config /etc/systemd/system/k3s.service.env
Just changed the hard-coded path in https://github.com/rancher/k3s/blob/master/k3s.service
EDIT: Of course: 0.10 doesn't work on Raspberry Pi (yet), so need to use the previous 0.9 ...
Don't think there was more to be done for packaging, so remaining is to build a variant of the iso.
Some more quirks when running k3s on Fedora:
Need to set up a CNI symlink:
/opt/cni/bin -> /usr/libexec/cni
When wanting to use CRI-O instead of containerd:
Edit the k3s.service and daemon-reload
...
[Service]
Type=notify
EnvironmentFile=/etc/systemd/system/k3s.service.env
ExecStart=/usr/bin/k3s server --container-runtime-endpoint /var/run/crio/crio.sock
...
Ultimately this should be configurable...
Got a new Raspberry Pi 4, but haven't done much with Tiny Core and aarch64 just yet.
Works fine with Ubuntu and Fedora though, the default Raspbian is still the same 32-bit.