Description:

It would very interessting to have k3s support in kamaji in addition to the "standard" kubernetes distributions.

As container you could use rancher/k3s:... similar to your k8s approach.

Thanks for opening a feature request @dirien, I appreciated that!

It required a bit of time to consider this feature, and I wanted to investigate a bit, and my assumptions could be wrong, so please, correct me if I'm wrong.

Components

k3s is a single binary that is scaffolding all the required components to run a minimalistic Kubernetes control plane: it starts the controller-manager, the scheduler, and obviously, the API server. The current design of Kamaji is expecting these components to be deployed as side-car containers of the Tenant Control Plane pod.

We could overcome this with the disabling components flags for the API Server container, just by setting with the CLI extra args the values --disable-scheduler and --disable-cloud-controller.

Default API Server flags

Kamaji is going to start an API Server with a set of default values that are mandatory for a vanilla Kubernetes cluster.

https://github.com/clastix/kamaji/blob/5e5bd83b6932fb1f5b24a624ec8d0e43380dcad8/internal/builders/controlplane/deployment.go#L527-L551

It seems most of them are not available in the CLI flags of k3s binary, and this seems a huge blocker to me.

Kine support

Since we're working on that via #22, I don't see a huge blocker.

Certificates

kl3s is creating the required certificates to expose itself and instantiate a communication with a secured tunnel to the datastore, and according to the docs, it seems not possible to block the generation of it.

We could check if providing the required certificates in a well-known path, this generation would be skipped: need help here understanding if this is possible.

Tag structure

The rollout of new versions is done by changing the version of the container image defined in the Tenant Control Plane specification.

apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: dirien
  namespace: default
spec:
  kubernetes:
    version: v1.23.1

It seems the k3s container images are not fully semantic-versioning compliant, like the v1.24.3-k3s1 (please, not the suffix -k3s1).

This would prevent any upgrade strategy from the Kamaji perspective since it's expecting a semantic version that is used to calculate also the upgrade path according to the kubeadm phases.

https://github.com/clastix/kamaji/blob/5e5bd83b6932fb1f5b24a624ec8d0e43380dcad8/internal/resources/kubeadm_upgrade.go#L108-L132

A possible solution could be a mirror with compliant semantic versioning: could be worth asking this directly to k3s maintainers?

---

Please, note that I'm not against this feature proposal, we're eager to increase the usage of Kamaji also for non-Vanilla Kubernetes distributions by getting insights I could have missed on the proposed distribution, or just by suggesting possible workaround and solutions to achieve the desired goal.

Going to close this due to inactivity, looking forward to having feedback in the future since we're open to supporting further distributions besides the vanilla one.