kind
kind copied to clipboard
Published
20 hours ago •
kubernetes-sigs
kubernetes-sigs
Updating to 0.20.0 causes kind workers to no longer be ready due to CNI errors
What happened:
After updating to version 0.20.0, our docker in docker like configuration no longer works. We are configured this way because we use Concourse as continuous integration and then are running kind containers inside of them. We are using the default kind config and the result is a control plane and two worker nodes. The worker nodes are stuck in a NotReady state:
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane 13m v1.27.3
kind-worker NotReady <none> 12m v1.27.3
kind-worker2 NotReady <none> 12m v1.27.3
When I describe one, I get:
Ready False Wed, 21 Jun 2023 19:14:46 +0000 Wed, 21 Jun 2023 19:14:46 +0000 KubeletNotReady [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful, container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized, CSINode is not yet initialized]
There is also a loop of: Normal Starting 1s kubelet Starting kubelet.
Here are the states of the pods:
kube-system coredns-5d78c9869d-66x7g 1/1 Running 0 19s
kube-system coredns-5d78c9869d-hs7bq 1/1 Running 0 19s
kube-system etcd-kind-control-plane 1/1 Running 0 35s
kube-system kindnet-6qm9x 0/1 Pending 0 15s
kube-system kindnet-hftnr 1/1 Running 0 19s
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 34s
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 31s
kube-system kube-proxy-dljl9 1/1 Running 0 19s
kube-system kube-proxy-krzqd 0/1 Pending 0 15s
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 31s
local-path-storage local-path-provisioner-6bc4bddd6b-t8jv8 1/1 Running 0 19s
supercolliderlite app-0 0/1 Pending 0 11s
What you expected to happen:
Control plane and work nodes get to a Ready state so the subsequent pods can get out of the Pending state and become Ready.
How to reproduce it (as minimally and precisely as possible):
This could be tricky because it is in a Concourse environment. What I have done is heavily based off of https://github.com/vmware-archive/kind-on-c
I pulled the scripts I needed to my own environment and use the dockerfile there to build my own image. I only needed to change my paths in the entrypoint script to use my local environment instead of that repo structure.
Anything else we need to know?:
I have exported the kind logs and there are a few potential issues I see but I am not sure where to start. I just highlighted the lines that seem to be erroring.
Inside serial.log:
-.slice: Failed to migrate controller cgroups from , ignoring: Input/output error
Failed to start Load Kernel Modules.
Inside kubelet.log and journal.log
failed to initialize top level QOS containers: root container [kubelet kubepods] doesn't exist
Inside containerd.log:
level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
level=info msg="No cni config template is specified, wait for other system components to drop the config."
Environment:
-
kind version: (use
kind version): kind v0.20.0 go1.20.4 linux/amd64 -
Runtime info: (use
docker infoorpodman info):
- Client: Docker Engine - Community
Version: 24.0.2
Context: default
Debug Mode: false
Server:
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 1
Server Version: 24.0.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 4.15.0-76-generic
Operating System: Ubuntu 22.04.2 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.41GiB
Name: 8fe3ad98-bf85-47c3-6ea2-4acf0415c776
ID: 4f05b697-ac93-4f0a-b997-2d34d0d37cc0
Docker Root Dir: /scratch/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
-
OS (e.g. from
/etc/os-release): UBUNTU_CODENAME=jammy -
Kubernetes version: (use
kubectl version):Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 -
Any proxies or other special environment settings?: Again, this environment is special because the issue only happens when running in concourse and therefore with a docker in docker setup. We did not have this issue on version 0.19.0 so I am not sure if this is a bug or if we now need to configure something new. Any help would be greatly appreciated as I have been trying to narrow this down all week! Thanks!!!
Facing the same issue but without a docker in docker configuration. Working on a machine with debian bookworm distro installed and have been successfully creating kind clusters till as of 3 weeks ago. Tried rolling back to Kubernetes version 1.26 image from 1.27 (latest) and facing the same issue.
Have attached the debug logs from kind export logs command. Happy to open a new issue if that would be better.
What you expected to happen: Expected the cluster to startup normally.
How to reproduce it (as minimally and precisely as possible):
Starting up a cluster with the default config i.e. kind create cluster surfaces the issue - log files for this attached as "attempt_3.tar.gz".
Anything else we need to know?: Kubelet startup fails with
Jul 09 16:51:07 kind-control-plane kubelet[4752]: E0709 16:51:07.083836 4752 kubelet.go:1480] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubelet kubepods] doesn't exist"
Jul 09 16:51:07 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 16:51:07 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.
Environment:
- kind version: (use
kind version):
$ kind version
kind v0.20.0 go1.20.4 linux/amd64
- Runtime info: (use
docker infoorpodman info):
$ docker info
Client: Docker Engine - Community
Version: 24.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.19.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 24.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 6.1.0-10-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.53GiB
Name: flagship
ID: ee7ea507-2ef0-47a1-be69-814f306e3722
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
- OS (e.g. from
/etc/os-release):
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
- Kubernetes version: (use
kubectl version): - reproduced with v1.27 and v1.26 kind images
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
The connection to the server localhost:8080 was refused - did you specify the right host or port?
- Any proxies or other special environment settings?: None
I experienced similar on kind node 1.27.0, but bumping to 1.27.3 fixed it for me (kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72)
I experienced a somewhat similar issue after I disabled cgroupsv2 (i.e., Cgroup Version: 1) on the host OS. I resolved my issue by switching to an older version of kind (0.17.0).
Workers can't join the cluster in GitLab with dind setup, too. I believe the issue might be related to the switch from ubuntu to debian for kindest/node base image in v0.19, as this looks like the biggest change. Downgrading to v0.18 is a workaround, because it still supports k8s v1.27 images.
kind-config.yaml:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
Output of kind create cluster:
kind create cluster --config=./e2e/kind-config.yaml
Creating cluster "kind" ...
• Ensuring node image (kindest/node:v1.27.3) 🖼 ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
• Preparing nodes 📦 📦 📦 📦 ...
✓ Preparing nodes 📦 📦 📦 📦
• Writing configuration 📜 ...
✓ Writing configuration 📜
• Starting control-plane 🕹️ ...
✓ Starting control-plane 🕹️
• Installing CNI 🔌 ...
✓ Installing CNI 🔌
• Installing StorageClass 💾 ...
✓ Installing StorageClass 💾
• Joining worker nodes 🚜 ...
✗ Joining worker nodes 🚜
Deleted nodes: ["kind-control-plane" "kind-worker2" "kind-worker3" "kind-worker"]
ERROR: failed to create cluster: failed to join node with kubeadm: command "docker exec --privileged kind-worker2 kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6" failed with error: exit status 1
Command Output: I0822 09:42:32.835954 213 join.go:412] [preflight] found NodeName empty; using OS hostname as NodeName
I0822 09:42:32.836000 213 joinconfiguration.go:76] loading configuration from "/kind/kubeadm.conf"
I0822 09:42:32.837113 213 controlplaneprepare.go:225] [download-certs] Skipping certs download
I0822 09:42:32.837139 213 join.go:529] [preflight] Discovering cluster-info
I0822 09:42:32.837157 213 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "kind-control-plane:6443"
I0822 09:42:32.867900 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 30 milliseconds
I0822 09:42:32.869030 213 token.go:223] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0822 09:42:39.160546 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 1 milliseconds
I0822 09:42:39.160738 213 token.go:223] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again
I0822 09:42:45.516439 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 2 milliseconds
I0822 09:42:45.517099 213 token.go:105] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "kind-control-plane:6443"
I0822 09:42:45.517123 213 discovery.go:52] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I0822 09:42:45.517132 213 join.go:543] [preflight] Fetching init configuration
I0822 09:42:45.517136 213 join.go:589] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I0822 09:42:45.550703 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s 200 OK in 33 milliseconds
I0822 09:42:45.555638 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy?timeout=10s 200 OK in 3 milliseconds
I0822 09:42:45.557162 213 kubelet.go:74] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
I0822 09:42:45.558558 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config?timeout=10s 200 OK in 1 milliseconds
I0822 09:42:45.560377 213 interface.go:432] Looking for default routes with IPv4 addresses
I0822 09:42:45.560395 213 interface.go:437] Default route transits interface "eth0"
I0822 09:42:45.560521 213 interface.go:209] Interface eth0 is up
I0822 09:42:45.560564 213 interface.go:257] Interface "eth0" has 3 addresses :[172.18.0.2/16 fc00:f853:ccd:e793::2/64 fe80::42:acff:fe12:2/64].
I0822 09:42:45.560582 213 interface.go:224] Checking addr 172.18.0.2/16.
I0822 09:42:45.560588 213 interface.go:231] IP found 172.18.0.2
I0822 09:42:45.560601 213 interface.go:[263](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L263)] Found valid IPv4 address 172.18.0.2 for interface "eth0".
I0822 09:42:45.560606 213 interface.go:443] Found active IP 172.18.0.2
I0822 09:42:45.565432 213 kubelet.go:121] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I0822 09:42:45.566024 213 kubelet.go:136] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I0822 09:42:45.566364 213 loader.go:373] Config loaded from file: /etc/kubernetes/bootstrap-kubelet.conf
I0822 09:42:45.566639 213 kubelet.go:157] [kubelet-start] Checking for an existing Node in the cluster with name "kind-worker2" and status "Ready"
I0822 09:42:45.568919 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:45.569220 213 kubelet.go:172] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I0822 09:42:46.651842 213 loader.go:373] Config loaded from file: /etc/kubernetes/kubelet.conf
I0822 09:42:46.652508 213 cert_rotation.go:137] Starting client certificate rotation controller
I0822 09:42:46.652837 213 loader.go:373] Config loaded from file: /etc/kubernetes/kubelet.conf
I0822 09:42:46.653206 213 kubelet.go:220] [kubelet-start] preserving the crisocket information for the node
I0822 09:42:46.653229 213 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///run/containerd/containerd.sock" to the Node API object "kind-worker2" as an annotation
I0822 09:42:47.177922 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 23 milliseconds
I0822 09:42:47.655940 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:48.155643 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:48.656832 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:49.155660 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:49.656153 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:50.156027 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:50.655937 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:51.155622 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 1 milliseconds
I0822 09:42:51.656754 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:52.155665 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:52.656568 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:53.156690 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:53.655449 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:54.156361 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:54.657274 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 3 milliseconds
I0822 09:42:55.156379 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:55.656291 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:56.156[269](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L269) 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:56.656307 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:57.156457 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:57.656533 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:58.156514 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:58.656514 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:59.156634 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:42:59.656667 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:00.156612 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:00.656745 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:01.155773 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:01.655755 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:02.155723 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:02.655748 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:03.155887 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:03.655729 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:04.155545 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:04.656491 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:05.156500 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:05.656415 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:06.156424 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:06.656649 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:07.156734 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:07.655720 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:08.155851 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:08.655686 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:09.156685 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:09.655608 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:10.156615 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:10.656488 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:11.156481 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:11.656426 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:12.156424 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:12.656[271](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L271) 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:13.156242 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:13.656359 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:14.156185 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:14.656092 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:15.156139 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:15.655867 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:16.155844 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:16.655951 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:17.156105 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:17.656122 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:18.156116 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:18.656067 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:19.156136 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:19.655800 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:20.155713 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:20.655734 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:21.155733 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:21.656846 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:22.155674 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:22.656629 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:23.156196 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:23.656100 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:24.155997 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:24.655946 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:25.155821 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0822 09:43:25.656[280](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L280) 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:26.155927 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:26.655909 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:27.156054 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:27.655914 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:28.155642 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:28.656535 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:29.156579 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:29.656556 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:30.156593 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0822 09:43:30.656422 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:31.156456 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:31.656582 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:32.156623 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:32.656756 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:33.156053 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:33.656108 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:34.156165 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:34.655768 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:35.155812 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:35.655755 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:36.155741 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:36.655694 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:37.156653 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:37.656870 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:38.155755 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:38.655906 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:39.155725 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:39.655519 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:40.156458 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0822 09:43:40.655670 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:41.156044 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:41.656104 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:42.155960 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:42.655962 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:43.155947 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:43.656181 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:44.156058 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:44.655849 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:45.155779 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:45.656673 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:46.156494 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:46.656573 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:47.156572 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:47.656577 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:48.157598 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 3 milliseconds
I0822 09:43:48.656884 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:49.156073 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:49.657451 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 3 milliseconds
I0822 09:43:50.156494 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:50.656606 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:51.156937 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:51.656049 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:52.155845 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:52.656181 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:53.156382 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:53.656358 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:54.156365 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:54.656361 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:55.156419 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:55.656611 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:56.156711 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:56.655909 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:57.155889 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:57.655895 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:58.155772 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:58.655611 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:59.156621 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:43:59.656583 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:00.156876 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0822 09:44:00.656088 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:01.156013 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:01.655951 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:02.155847 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:02.674207 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 20 milliseconds
I0822 09:44:03.155778 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:03.675797 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 22 milliseconds
I0822 09:44:04.155716 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:04.655616 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:05.156721 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:05.655691 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:06.155726 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:06.655746 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:07.155668 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:07.656722 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:08.155686 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:08.656682 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:09.156871 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:09.655621 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:10.156672 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:10.655793 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:11.155709 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:11.655646 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:12.156857 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:12.655458 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 1 milliseconds
I0822 09:44:13.156993 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:13.655877 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:14.155866 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:14.655932 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:15.155824 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:15.656039 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:16.155844 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:16.656215 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:17.156064 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:17.656102 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:18.155623 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:18.656567 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:19.156494 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:19.656[324](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L324) 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:20.156373 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:20.656377 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:21.156462 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:21.656494 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:22.156398 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:22.656455 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:23.156977 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:23.655918 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:24.156066 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:24.656220 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:25.156396 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:25.656353 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:26.156362 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:26.656378 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:27.156624 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:27.656754 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:28.155557 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:28.656450 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:29.156487 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:29.656517 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:30.156703 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:30.656811 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:31.155749 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:31.655642 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:32.156535 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:32.656638 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:33.156663 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:33.656655 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:34.156612 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:34.656701 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:35.156526 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:35.656525 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:36.156390 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:36.656385 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:37.156818 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:37.655729 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:38.155541 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:38.656421 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:39.156392 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:39.656386 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:40.156347 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0822 09:44:40.656[336](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L336) 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:41.156006 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:41.655968 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:42.155830 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:42.655802 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:43.155917 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:43.655679 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:44.155721 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:44.655655 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:45.156687 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:45.656591 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:46.156578 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:46.655670 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s 404 Not Found in 2 milliseconds
I0822 09:44:46.657420 213 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/nodes/kind-worker2?timeout=10s [404](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L404) Not Found in 1 milliseconds
nodes "kind-worker2" not found
error uploading crisocket
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletStartJoinPhase
cmd/kubeadm/app/cmd/phases/join/kubelet.go:222
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
cmd/kubeadm/app/cmd/phases/workflow/runner.go:[446](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L446)
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
cmd/kubeadm/app/kubeadm.go:50
main.main
cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
error execution phase kubelet-start
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
cmd/kubeadm/app/cmd/phases/workflow/runner.go:260
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1
cmd/kubeadm/app/cmd/join.go:179
github.com/spf13/cobra.(*Command).execute
vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
cmd/kubeadm/app/kubeadm.go:50
main.main
cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1[598](https://gitlab.com/castai/kubecast/-/jobs/4919916274#L598)
These are the logs of a worker node that fails to join the cluster:
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v1
INFO: detected cgroupns
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: detected IPv4 address: 172.18.0.2
INFO: detected IPv6 address: fc00:f853:ccd:e793::2
INFO: starting init
systemd 247.3-7+deb11u2 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 11 (bullseye)!
Set hostname to <kind-worker3>.
Failed to attach 1 to compat systemd cgroup /init.scope: No such file or directory
Couldn't move remaining userspace processes, ignoring: Input/output error
Queued start job for default target Graphical Interface.
-.slice: Failed to migrate controller cgroups from , ignoring: Input/output error
[ OK ] Created slice slice used to run Kubernetes / Kubelet.
system.slice: Failed to migrate controller cgroups from /system.slice, ignoring: Input/output error
[ OK ] Created slice system-modprobe.slice.
[ OK ] Started Dispatch Password …ts to Console Directory Watch.
[ OK ] Set up automount Arbitrary…s File System Automount Point.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Reached target Paths.
[ OK ] Reached target Slices.
[ OK ] Reached target Swap.
[ OK ] Listening on Journal Audit Socket.
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Listening on Journal Socket.
[ OK ] Reached target Sockets.
Failed to attach 152 to compat systemd cgroup /dev-hugepages.mount: No such file or directory
Mounting Huge Pages File System...
Failed to attach 152 to compat systemd cgroup /dev-hugepages.mount: No such file or directory
Failed to attach 153 to compat systemd cgroup /sys-kernel-debug.mount: No such file or directory
Mounting Kernel Debug File System...
Failed to attach 153 to compat systemd cgroup /sys-kernel-debug.mount: No such file or directory
Failed to attach 154 to compat systemd cgroup /sys-kernel-tracing.mount: No such file or directory
Mounting Kernel Trace File System...
Failed to attach 154 to compat systemd cgroup /sys-kernel-tracing.mount: No such file or directory
Starting Load Kernel Module configfs...
Starting Load Kernel Module fuse...
systemd-journald.service: Failed to migrate controller cgroups from /system.slice/systemd-journald.service, ignoring: Input/output error
Failed to attach 157 to compat systemd cgroup /system.slice/systemd-journald.service: No such file or directory
Starting Journal Service...
Failed to attach 157 to compat systemd cgroup /system.slice/systemd-journald.service: No such file or directory
Starting Load Kernel Modules...
Starting Remount Root and Kernel File Systems...
[ OK ] Mounted Huge Pages File System.
[ OK ] Mounted Kernel Debug File System.
[ OK ] Mounted Kernel Trace File System.
[email protected]: Succeeded.
[ OK ] Finished Load Kernel Module configfs.
[email protected]: Succeeded.
[ OK ] Finished Load Kernel Module fuse.
Failed to attach 160 to compat systemd cgroup /sys-fs-fuse-connections.mount: No such file or directory
Mounting FUSE Control File System...
Failed to attach 160 to compat systemd cgroup /sys-fs-fuse-connections.mount: No such file or directory
Failed to attach 161 to compat systemd cgroup /sys-kernel-config.mount: No such file or directory
Mounting Kernel Configuration File System...
Failed to attach 161 to compat systemd cgroup /sys-kernel-config.mount: No such file or directory
systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
systemd-modules-load.service: Failed with result 'exit-code'.
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[ OK ] Finished Remount Root and Kernel File Systems.
[ OK ] Mounted FUSE Control File System.
[ OK ] Mounted Kernel Configuration File System.
Starting Apply Kernel Variables...
Starting Create System Users...
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Finished Apply Kernel Variables.
[ OK ] Finished Update UTMP about System Boot/Shutdown.
[ OK ] Finished Create System Users.
Starting Create Static Device Nodes in /dev...
[ OK ] Finished Create Static Device Nodes in /dev.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
[ OK ] Started Journal Service.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Reached target Basic System.
[ OK ] Reached target Timers.
Starting Flush Journal to Persistent Storage...
Starting Undo KIND mount hacks...
[ OK ] Finished Undo KIND mount hacks.
Starting containerd container runtime...
[ OK ] Finished Flush Journal to Persistent Storage.
[ OK ] Started containerd container runtime.
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Starting Update UTMP about System Runlevel Changes...
[ OK ] Finished Update UTMP about System Runlevel Changes.
And the details of the failing Load Kernel Modules systemd service:
/ # docker exec c8f41c1402c8 systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static)
Active: failed (Result: exit-code) since Tue 2023-08-22 10:00:09 UTC; 24s ago
Docs: man:systemd-modules-load.service(8)
man:modules-load.d(5)
Main PID: 158 (code=exited, status=1/FAILURE)
Warning: journal has been rotated since unit was started, output may be incomplete.
/ # docker exec c8f41c1402c8 journalctl _PID=158
-- Journal begins at Tue 2023-08-22 10:00:09 UTC, ends at Tue 2023-08-22 10:01:25 UTC. --
Aug 22 10:00:09 kind-worker3 systemd-modules-load[158]: Failed to look up module alias 'iscsi_tcp': Function not implemented
Aug 22 10:00:09 kind-worker3 systemd-modules-load[158]: Failed to look up module alias 'ib_iser': Function not implemented
We've made a lot of changes since v0.18, did you get a chance to test v0.20?
I experienced a somewhat similar issue after I disabled cgroupsv2 (i.e., Cgroup Version: 1) on the host OS. I resolved my issue by switching to an older version of kind (0.17.0).
In the future please file a separate bug report and fill out the template, there's not really enough info to respond here and this looks off-topic. However see also https://kind.sigs.k8s.io/docs/user/known-issues/#failure-to-create-cluster-with-cgroups-v2
Kubernetes only started supporting v2 in 1.19, but v2 is the path forward.
I experienced similar on kind node 1.27.0, but bumping to 1.27.3 fixed it for me (kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72)
please see the release notes. You should be using an image known to be compatible with the release. You can build an image at different k8s versions yourself if the desired version isn't available following the quick start.
I highly recommend avoiding an additional layer of container "nesting", we apply a lot of tricks in kind to get just one to work well enough with kubernetes. This is why kind is supplied as a portable statically linked go binary for easily using on the host.
I highly recommend avoiding an additional layer of container "nesting", we apply a lot of tricks in kind to get just one to work well enough with kubernetes. This is why kind is supplied as a portable statically linked go binary for easily using on the host.
kind works in a dind setup, the main control-plane node also boots up fine, but additional workers fail to join the cluster. Are there any docs on how to run and join additional workers without using containers?
kind works in a dind setup, the main control-plane node also boots up fine, but additional workers fail to join the cluster. Are there any docs on how to run and join additional workers without using containers?
no, this is out of scope and unsupported.
@BenTheElder @saumas To be clear, this was working with kind 0.19.0 and only started breaking with 0.20.0. The version of kubernetes does not seem to matter. Im not sure how to get past this or even debug what is actually going on here
@xtreme-shane-lattanzio
Try using cgroup v2 as detailed in https://github.com/kubernetes-sigs/kind/issues/3372#issuecomment-1739937686
@mboutet I ran grep cgroup /proc/mounts in my docker image and the output is cgroup /sys/fs/cgroup cgroup2 ro,nosuid,nodev,noexec,relatime 0 0 I think that means v2 is already being used right?
@xtreme-shane-lattanzio not necessarily and from the docker info output in the issue description:
Cgroup Version: 1
you can see that cgroup v1 is used by docker.
afaik, grepping will only show you if cgroup v2 is available on your system, not necessarily if docker is using it.
@mboutet Thanks for the info. I ran it on my local machine and also as Cgroup Version: 2. I am running this setup in concourse so I am wondering if there is something in their setup that is saying v1. Or maybe the kind-on-c think that I am using is somehow using the wrong version?
@xtreme-shane-lattanzio I can't really help you regarding your concourse scenario, but it might be the case that the nodes on which the concourse jobs are running also need to have cgroup v2 enabled by default.
@mboutet My scenario is really just a docker in docker case. When running this, docker info is showing cgroup v1. This is the repo and script I was using. Since it is archived, I pulled what I needed to my own and am trying to play around with it. Based on your suggestion have a feeling that something is wrong around line 20 in this script: https://github.com/vmware-archive/kind-on-c/blob/master/entrypoint.sh Or maybe I just need to add something in here to enable v2? I tried to follow the instructions you posted but I do not even have grub or related files available so I am not sure what I am doing wrong. Either way Im still not sure that it would solve my issue here but Im willing to try anything!
@xtreme-shane-lattanzio, I also have the same use cases as you. Locally, we use dev containers to develop K8s tooling and operators and we run kind within that dev container. In our CI, we have containerized ephemeral GitHub Actions runners on which we run automated integration tests also using kind.
The key is really to enable cgroup v2 at the host level. Remember that a container doesn't have its own kernel. It "shares" it with the host system. Therefore, you really need to enable this on the host itself. Then your DinD containers will also see it.
I can confirm, switching to cgroup v2 fixed the issues for us.
We were running kind for integration tests in CI on top of EKS and Docker in Docker.
After we switched our Kubernetes nodes AMI from AL2 to AL2023 with cgroup v2 support, the issues were resolved.
