Support sysbox installation on K3s?

[Teachh]

Hi everyone, I am actually trying to install Sysbox in my Kubernetes Cluster with all the nodes on 23.04.

I saw that the last supported version for Ubuntu is 22.04. Is there any way to use the DaemonSet from this manifests?

I tried to change the Dockerimage from the DaemonSet to allow my node version but I'm having the following error in the Pod

Job for kubelet-config-helper.service failed because the control process exited with error code. See "systemctl status kubelet-config-helper.service" and "journalctl -xe" for details.

Thanks!

[ctalledo]

Hi @Teachh, thanks for giving Sysbox a try.

If I understand correctly, you are trying to install Sysbox on a K8s cluster with nodes using Ubuntu 23.04 (rather than Ubuntu 22.04 for example).

Yes, we don't officially support sysbox-deploy-k8s on Ubuntu 23.04, but having said that I am surprised it failed after you modified the distro version check.

Question: when it failed, did you try logging into the K8s node and typing systemctl status kubelet-config-helper.service to see why it failed?

Also, what version of K8s are you running?

Thanks!

[Teachh]

Hi @ctalledo, first of all thanks for answering!

Executing the systemctl command output is:

systemctl status kubelet-config-helper.service
× kubelet-config-helper.service - Kubelet config service
     Loaded: loaded (/lib/systemd/system/kubelet-config-helper.service; static)
     Active: failed (Result: exit-code) since Wed 2024-10-16 12:55:08 UTC; 7s ago
    Process: 4036663 ExecStart=/bin/sh -c /usr/local/bin/kubelet-config-helper.sh (code=exited, status=1/FAILURE)
   Main PID: 4036663 (code=exited, status=1/FAILURE)
        CPU: 39ms

Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: + kubelet_bin=
Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: + '[' -z '' ']'
Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: + die 'Kubelet binary not identified.'
Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: + msg='Kubelet binary not identified.'
Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: + echo 'ERROR: Kubelet binary not identified.'
Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: ERROR: Kubelet binary not identified.
Oct 16 12:55:08 k8s-slave2-gotham sh[4036664]: + exit 1
Oct 16 12:55:08 k8s-slave2-gotham systemd[1]: kubelet-config-helper.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 12:55:08 k8s-slave2-gotham systemd[1]: kubelet-config-helper.service: Failed with result 'exit-code'.
Oct 16 12:55:08 k8s-slave2-gotham systemd[1]: Failed to start kubelet-config-helper.service - Kubelet config service.

The version of all the nodes of the cluster is v1.28.2+k3s1 with a K3S distribution.

To bypass the version check, I have modified this file and add my Ubuntu version on lines 709 and 263.

Thanks!

[ctalledo]

Hi @Teachh,

Thanks for the info, that clarifies the problem, and it's not related to the version of Ubuntu.

Rather, sysbox-deploy-k8s only works on K8s cluster, not on k3s clusters because the latter does not carry a separate binary for the kubelet component which sysbox-deploy-k8s reconfigures and restarts (to point the kubelet to CRI-O instead of containerd).

I am sure we could make it work on k3s, but it's a fair amount of work and there are no plans (but of course if anyone wants to contribute this we would be happy to take a look).

[Teachh]

Hi @ctalledo.

Perfect then, since this is something that could benefit a lot to our infrastructure I will take a look and update the improvements through this issue if possible.

Thanks!

[GrigoriOH]

Hi, thank you for clarification on this topic. We waited for 1.30 support, installed sysbox on our systems and just stumbled upon the k3s restraint. Here, I just wanted to voice our interest in k3s support, give other users the possibility to 👍 their need and maybe ask for a mention of this constraint in the docs :)

Bests and thanks for your work!

[Teachh]

Hi @GrigoriOH and @ctalledo ,

I have gotten sysbox working on k3s. You need to install CRI-O and have a version strictly below 1.30. In my case I am running it in Ubuntu 22.04. I created this repo to reproduce my steps. Let me know if you need help :)

[Teachh]

Hi,

I changed the script file into a Ansible playbook to make it easy in this repo.

[ctalledo]

Hi @Teachh, great to hear you got it to work by installing CRI-O + Sysbox on the K3s node. I'll take a closer look to see how you did it.

What would be great (and not sure it's possible yet), is to have sysbox-deploy-k8s work on a K3s cluster, where it does the installation of CRI-O and Sysbox. Do you think this is possible based on your investigations?

[Teachh]

Hi @Teachh, great to hear you got it to work by installing CRI-O + Sysbox on the K3s node. I'll take a closer look to see how you did it.

What would be great (and not sure it's possible yet), is to have sysbox-deploy-k8s work on a K3s cluster, where it does the installation of CRI-O and Sysbox. Do you think this is possible based on your investigations?

The short answer is yes, it is possible.

What I have done is installing everything manually in the Node instead of using the sysbox-deploy-k8s because of some errors related with Sysbox Installation. To solve that I installed Sysbox Manually with this Tasks, once Sysbox was installed manually everything was working as expected :)

I did not investigate further how the Daemon works but if you need help with that let me know!

[ctalledo]

Thanks @Teachh, I took a look at the Ansible playbooks.

Question: where is it that you configure the Kubelet to use CRI-O?

[Teachh]

Thanks @Teachh, I took a look at the Ansible playbooks.

Question: where is it that you configure the Kubelet to use CRI-O?

Hi @ctalledo,

I do it replacing the k3s-agent.service configuration with this file. After CRI-O is installed, first Sysbox task replaces the file.

[ctalledo]

I see it, thanks for sharing @Teachh!

[johnwmail]

Hi all,

Is it possible , install sysbox as the container runtime with k3s single node? Thanks.

[Teachh]

Hi @johnwmail ,

I have done it time ago before implementing sysbox to my prod environment and worked as expected. Not sure if changing containerd to CRI-O can affect to the performance of the control-pane. Did not see any test or graphs related with this.

[johnwmail]

I have done it time ago before implementing sysbox to my prod environment and worked as expected.

Would you share how you did it? Since a lot of people run k3s as an single node k8s. Thanks.

[reski-rukmantiyo]

any solutions for this? been searching solution for almost a day, but still havent got solution for k3s single node with sysbox.