Installing Sysbox on K8s clusters without using sysbox-k8s-deploy

[TomasKohout]

Hi guys,

I've tried to install CRI-O and sysbox without the use of daemonset installation because it's not viable for us to have such installation method and I was unable to do it without going through your installation scripts where, at the end, I've found out that I need to use your package of CRI-O. Do you have an idea where will your changes land in upstream?

Would it be possible to create documentation for guys like me who can't use daemonset installation?

Another thing is that I'm unable to install sysbox deb package when reinstalling node through PXE because postinstall script in dep package will complain that kernel module is missing although it is present when node reboots.

Thank you very much. :slightly_smiling_face:

[ctalledo]

Hi @TomasKohout, thanks for reaching out and for your interest in Sysbox.

Responding to your questions:

I've found out that I need to use your package of CRI-O. Do you have an idea where will your changes land in upstream?

That's correct; we made a minor fix to CRI-O to ensure it never maps the container's root user to the host's root user. We have communicated this change to the CRI-O maintainers, but have not had a chance to upstream it yet (it requires a bit more work on our side to upstream it). We hope to get this done in the coming 1->2 months, though it's not a top priority at this time.

Would it be possible to create documentation for guys like me who can't use daemonset installation?

We don't have plans to do this because the installation steps are rather complex and depend on the K8s distro. This is why we scripted them into the sysbox-deploy-k8s daemonset.

Having said this, if this is for an enterprise use case, we may be able to offer such a service via a Sysbox-EE subscription.

Another thing is that I'm unable to install sysbox deb package when reinstalling node through PXE because postinstall script in dep package will complain that kernel module is missing although it is present when node reboots.

Could you please show us the error message that you get?

Thanks!

[TomasKohout]

Hi @ctalledo, thank you for your response. :slightly_smiling_face:

That's correct; we made a minor fix to CRI-O to ensure it never maps the container's root user to the host's root user. We have communicated this change to the CRI-O maintainers, but have not had a chance to upstream it yet (it requires a bit more work on our side to upstream it). We hope to get this done in the coming 1->2 months, though it's not a top priority at this time.

I'm looking forward to this. :slightly_smiling_face:

We don't have plans to do this because the installation steps are rather complex and depend on the K8s distro. This is why we scripted them into the sysbox-deploy-k8s daemonset.

Understood. The issue is that we use our own bare metal servers and I don't want to use daemonset approach because it's a big change for our setup.

Having said this, if this is for an enterprise use case, we may be able to offer such a service via a Sysbox-EE subscription.

Noted. :+1:

Could you please show us the error message that you get?

The postinst script contains check check_configfs_loaded for configfs kernel module and if the module is not loaded, posinst will fail.

I've tried to do steps that you do within daemonset installation by my own, but for some reason, cotnainers fail to start with this message. Could you point me to the right direction please?

time="2022-02-01T17:05:57Z" level=error msg="failed to write \"0-3,5,7,9-55,57,59,61-103\": write /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad75417e_e08d_40a1_8722_b12e279f3d73.slice/crio-main.scope/cpuset.cpus: device or resource busy"

[ctalledo]

Hi @TomasKohout,

The issue is that we use our own bare metal servers and I don't want to use daemonset approach because it's a big change for our setup.

Just curious, why is that (if you can share this info)? I ask because generally the sysbox-deploy-k8s daemonset is the easiest approach to installing Sysbox on a cluster, whether it's on VMs or bare-metal.

The postinst script contains check check_configfs_loaded for configfs kernel module and if the module is not loaded, posinst will fail.

@rodnymolina is working on fixing this for our upcoming release (Feb). I think the Sysbox installer's requirement for configfs is too stringent, needs relaxing.

[samcday]

I'd like to +1 @TomasKohout feeling on this.

I tried to use the installer just now on a baremetal cluster (Ubuntu 20.04, containerd runtime, nothing too fancy but not necessarily "vanilla") and ran into #525. It probably would have been smoother if I were to do a "ground up" install. i.e get sysbox runtime + CRI-O installed and running, and then test them directly before switching kubelet to use them.

IMO whilst it makes lots of sense to script the happy path for the EKS/AKS/$$bigcloud$$ offerings, it makes less sense for baremetal clusters which often tend to be standard but with tiny little deviations here and there.

[rodnymolina]

@samcday, thanks for your feedback, I understand your point. The reason that we created this daemon-set was to simplify the somewhat complex process of installing Sysbox + CRI-O in a running K8s cluster, especially when we take into consideration the large number of existing K8s distributions and the variety of deployment patterns utilized by cloud vendors.

In regards to your goal of "testing before switching kubelet", please keep in mind that, on a given k8s node, kubelet can operate with a single CRI at a time, being this one docker-shim, containerd, or cri-o, so at this level you won't really have the option of testing concurrent environments. Having said that, once sysbox-k8s-deploy does its installation magic, your existing pods (as well as the new ones) will continue to use the regular (oci)runc for their instantiation -- IOW, sysbox-pods require explicit configuration.

Another thing, at the risk of stating the obvious: you can always install Sysbox on your bare-metal servers through the standalone package installers and play with it through the docker or crictl interface, so you don't really need to install sysbox-k8s-deploy daemonset to test Sysbox itself.

Will get back to you on #525 later on.

[TomasKohout]

Hi guys, :slightly_smiling_face:

Just curious, why is that (if you can share this info)? I ask because generally the sysbox-deploy-k8s daemonset is the easiest approach to installing Sysbox on a cluster, whether it's on VMs or bare-metal.

we want to controll what is installed on our workers and what settings are set. Installation via daemonset destroys this transparency which makes sysbox unusable for us.

In the meantime, we have moved to containerd which is not supported by sysbox. :slightly_frowning_face:

[ctalledo]

Thanks @TomasKohout;

Sounds like documenting the manual installation process would solve the issue for you. Let me check if we can do that, at least for the case of vanilla K8s clusters and with the caveat of "this is not a fully supported installation method".

Note however that we've not yet upstreamed the patch we added to CRI-O, so this will require using the patched CRI-O in the Nestybox repo.

In the meantime, we have moved to containerd which is not supported by sysbox.

Got it; unfortunately containerd does not yet support deploying pods with the user-ns, and this is a must-have for Sysbox, which is why it relies on CRI-O currently.