[FEATURE] Support RKE2

[olljanat]

Describe the solution you'd like

As far I understand RKE2 is created based on K3s so it would be nice to have switch like --rke2-mode here which would use its images instead of and enable CIS hardenings automatically.

[iwilltry42]

Hi @olljanat , RKE2 works quite a bit differently than K3s and also takes different arguments/environment variables than K3s, so it would require quite some changes on k3d side. I'm not 100% familiar with all the differences, but it would certainly be a nice thing to have in k3d. I'd happily accept any PR introducing this feature :)

[olljanat]

As far I understand similar image than which k3d is using with K3s can be build using https://github.com/rancher/rke2/blob/release-1.21/Dockerfile but those are not automatically published anywhere currently (or at least I was not able to find them) so I did build latest version and published in on ollijanatuinen/rke2:v1.21-test

Simple example how to test:

# Create cluster
k3d cluster create rke2 --no-lb --image ollijanatuinen/rke2:v1.21-test --api-port 0.0.0.0:6443

# From another shell follow with this command until you see message "Handling backend connection request [k3d-rke2-server-0]"
docker logs -f k3d-rke2-server-0

# Fetch kubeconfig
docker exec -it k3d-rke2-server-0 cat /etc/rancher/rke2/rke2.yaml > .kube/config

# Test that you can see cluster nodes:
kubectl get nodes
NAME                STATUS   ROLES                       AGE     VERSION
k3d-rke2-server-0   Ready    control-plane,etcd,master   4m11s   v1.21.5+rke2r2

It does not use those K3S_ environment variables so that part definitely need some changes to k3d side too.

[brkirchmann]

@olljanat when I run the Create cluster steps above, I get the following:

DEBU[0002] Waiting for node k3d-rke2-server-0 to get ready (Log: 'k3s is up and running') TRAC[0002] NodeWaitForLogMessage: Node 'k3d-rke2-server-0' waiting for log message 'k3s is up and running' since '2021-11-16 19:46:49 +0000 UTC' ERRO[0003] Failed Cluster Start: Failed to start server k3d-rke2-server-0: Node k3d-rke2-server-0 failed to get ready: error waiting for log line k3s is up and running from node 'k3d-rke2-server-0': stopped returning log lines

I am assuming rke2 does not output a log with "k3s is up and running". How were you able to get this to start?

[olljanat]

I am assuming rke2 does not output a log with "k3s is up and running". How were you able to get this to start?

Yes that first console gets stuck waiting for something. That why instruction above says that you need test from another shell. That however should be minor thing to fix.

Much bigger question is that will Rancher/SUSE start providing those RKE2 images somewhere which are needed on here? Without it custom build is needed from each RKE2 version which does not make any sense.

[brkirchmann]

My deploy kept rolling back after the failure, so I added the --no-rollback flag

I do see the following now:

k3d cluster list
NAME   SERVERS   AGENTS   LOADBALANCER
rke2   1/1       0/0      false

Unfortunately, while I see the cluster, the node stays in a restarting state:

k3d node list
NAME                ROLE     CLUSTER   STATUS
k3d-rke2-server-0   server   rke2      restarting

I did a manual stop/start using k3d node stop/start, unfortunately it goes back to the restarting loop.

will keep working on it.

[brkirchmann]

ok, so the only way I was able to get the image deployed (ollijanatuinen/rke2:v1.21-test or rancher/rke2-runtime:v1.21.6-rc2-rke2r2) was to create a cluster:

> k3d cluster create rke2 --no-lb

then import the image

> k3d image import ollijanatuinen/rke2:v1.21-test -c rke2

INFO[0000] Importing image(s) into cluster 'rke2'
INFO[0000] Starting new tools node...
INFO[0001] Starting Node 'k3d-rke2-tools'
INFO[0003] Saving 1 image(s) from runtime...
INFO[0029] Importing images into nodes...
INFO[0029] Importing images from tarball '/k3d/images/k3d-rke2-images-20211116211228.tar' into node 'k3d-rke2-server-0'...
INFO[0053] Removing the tarball(s) from image volume...
INFO[0054] Removing k3d-tools node...
INFO[0056] Successfully imported image(s)
INFO[0056] Successfully imported 1 image(s) into 1 cluster(s)

this starts fine, but when I exec onto the docker image docker exec -it k3d-rke2-server-0 /bin/sh. There is no /etc/rancher/rke2/rke2.yaml. The only kube config file is under /etc/rancher/k3s/k3s.yaml which is the default one form a normal cluster creation. This leads me to believe this did not deploy this image properly or at least like you were able to deploy it. Any advice?

[iwilltry42]

@brkirchmann , what you did there is importing the RKE2 image into the K3s cluster's containerd. What @olljanat did was creating the cluster itself with RKE2 instead of K3s. This comment holds all the information. However, I guess there are quite a few settings that need to be adapted on k3d side first to get RKE2 to start properly. It's on the roadmap :+1:

[brkirchmann]

Right, unfortunately I could not get what @olljanat did to work for me. It just fails out and stops, it doesn't stay up and running at all as noted in https://github.com/rancher/k3d/issues/783#issuecomment-970646921

[mattmattox]

This would be very helpful for Rancher Support to allow us to test a wide range of RKE2 versions as part of our pipelines.