unable to install k3s and Fleet on Elemental

[Prasadrpatil]

I have created ISO by elemental-toolkit method - Link

I have deployed K3S + Fleet + System Upgrade Controller over a Elemental using cloud-init yaml configuration file - Link

stages:
   rootfs.after:
     - if: '[ -f "/run/cos/recovery_mode" ]'
       name: "Repart image"
       layout:
         # It will partition a device including the given filesystem label or part label (filesystem label matches first)
         device:
           label: COS_RECOVERY
         add_partitions:
           - fsLabel: COS_STATE
             # 15Gb for COS_STATE, so the disk should have, at least, 20Gb
             size: 15360
             pLabel: state
           - fsLabel: COS_PERSISTENT
             # unset size or 0 size means all available space
             pLabel: persistent
     - if: '[ ! -f "/run/cos/recovery_mode" ]'
       name: "Persistent state"
       environment_file: /run/cos/cos-layout.env
       environment:
         VOLUMES: "LABEL=COS_OEM:/oem LABEL=COS_PERSISTENT:/usr/local"
         OVERLAY: "tmpfs:25%"
         RW_PATHS: "/var /etc /srv"
         PERSISTENT_STATE_PATHS: "/root /opt /home /var/lib/rancher /var/lib/kubelet /etc/systemd /etc/rancher /etc/ssh"
   network.before:
     - name: "Setup SSH keys"
       authorized_keys:
         root:
         # It can download ssh key from remote places, such as github user keys (e.g. `github:my_user`)
         - my_custom_ssh_key
     - if: '[ ! -f "/run/cos/recovery_mode" ]'
       name: "Fleet deployment"
       files:
       - path: /etc/k3s/manifests/fleet-config.yaml
         content: |
              apiVersion: helm.cattle.io/v1
              kind: HelmChart
              metadata:
                name: fleet-crd
                namespace: kube-system
              spec:
                chart: https://github.com/rancher/fleet/releases/download/v0.8.0-rc.1/fleet-crd-0.8.0-rc.1.tgz
              ---
              apiVersion: helm.cattle.io/v1
              kind: HelmChart
              metadata:
                name: fleet
                namespace: kube-system
              spec:
                chart: https://github.com/rancher/fleet/releases/download/v0.8.0-rc.1/fleet-0.8.0-rc.1.tgz              
   network:
     - if: '[ -f "/run/cos/recovery_mode" ]'
       name: "Deploy cos-system"
       commands:
         # Deploys the recovery image.
         # use --docker-image to deploy a custom image
         # e.g. `elemental reset --docker-image quay.io/my_custom_repo:my_image`
         - elemental reset --reboot
     - if: '[ ! -f "/run/cos/recovery_mode" ]'
       name: "Setup k3s"
       directories:
       - path: "/usr/local/bin"
         permissions: 0755
         owner: 0
         group: 0
       commands:
       - |
            curl -sfL https://get.k3s.io | \
            INSTALL_K3S_VERSION="v1.27.4+k3s1" \
            INSTALL_K3S_EXEC="--tls-san {{.Values.node.hostname}}" \
            INSTALL_K3S_SELINUX_WARN="true" \
            sh -
            # Install fleet 
            kubectl apply -f /etc/k3s/manifests/fleet-config.yaml
            # Install system-upgrade-controller
            kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/v0.6.2/manifests/system-upgrade-controller.yaml  

I am able to confirm that k3s is running.

The issue is that Node is not registered with the cluster and Missing kubelet configuration. as a result all Key K3s components pods are in pending state.

journalctl -f -u k3s

Aug 22 09:28:28 elemental k3s[1459]: time="2023-08-22T09:28:28Z" level=info msg="certificate CN=elemental signed by CN=k3s-server-ca@1692696161: notBefore=2023-08-22 09:22:41 +0000 UTC notAfter=2024-08-21 09:28:28 +0000 UTC"
Aug 22 09:28:28 elemental k3s[1459]: time="2023-08-22T09:28:28Z" level=info msg="Waiting for control-plane node elemental startup: nodes \"elemental\" not found"
Aug 22 09:28:28 elemental k3s[1459]: time="2023-08-22T09:28:28Z" level=info msg="certificate CN=system:node:elemental,O=system:nodes signed by CN=k3s-client-ca@1692696161: notBefore=2023-08-22 09:22:41 +0000 UTC notAfter=2024-08-21 09:28:28 +0000 UTC"
Aug 22 09:28:29 elemental k3s[1459]: time="2023-08-22T09:28:29Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: invalid argument"
Aug 22 09:28:29 elemental k3s[1459]: time="2023-08-22T09:28:29Z" level=info msg="Waiting for control-plane node elemental startup: nodes \"elemental\" not found"
Aug 22 09:28:29 elemental k3s[1459]: E0822 09:28:29.630593    1459 resource_quota_controller.go:441] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1
Aug 22 09:28:30 elemental k3s[1459]: W0822 09:28:30.192315    1459 garbagecollector.go:816] failed to discover some groups: map[metrics.k8s.io/v1beta1:stale GroupVersion discovery: metrics.k8s.io/v1beta1]
Aug 22 09:28:30 elemental k3s[1459]: time="2023-08-22T09:28:30Z" level=info msg="Waiting for control-plane node elemental startup: nodes \"elemental\" not found"

[kkaempf]

Not sure if this a supported method. 🤔

Is https://elemental.docs.rancher.com an alternative for you ?

[Prasadrpatil]

Is there any way we can install k3s independent of rancher on elemental?

[kkaempf]

That wasn't tried before afaik, you're on your own I'm afraid.

You probably have to check the complete k3s log and look for errors.

There's also an #elemental channel on rancher-users.slack.com where more people are listening 🤞🏻