Unable to join nodes to cluster using microk8s join command

[bhargavsangabattula]

Hi All, I am in process of setting of new microk8s cluster with 3 nodes which are running on Ubuntu 22.04 and kernel version is 6.5.0-1017-aws on AWS infra everytime I am getting the this error while joining the nodes to primary node to form the cluster and seeing this error when run the microk8s.status on secondary nodes. ubuntu@ip-172-31-37-61:~$ microk8s join 172.31.44.251:25000/f7159c7215fe49c70b71c79545b5c584/f496dd46d0b8 Contacting cluster at 172.31.44.251 Waiting for this node to finish joining the cluster. .. .. .. .. .. .. .. .. .. ..
ubuntu@ip-172-31-37-61:~$ microk8s.status microk8s is not running. Use microk8s inspect for a deeper inspection. ubuntu@ip-172-31-37-61:~$ microk8s.inspect Inspecting system Inspecting Certificates Inspecting services Service snap.microk8s.daemon-cluster-agent is running Service snap.microk8s.daemon-containerd is running Service snap.microk8s.daemon-kubelite is running Service snap.microk8s.daemon-k8s-dqlite is running Service snap.microk8s.daemon-apiserver-kicker is running Copy service arguments to the final report tarball Inspecting AppArmor configuration Gathering system information Copy processes list to the final report tarball Copy disk usage information to the final report tarball Copy memory usage information to the final report tarball Copy server uptime to the final report tarball Copy openSSL information to the final report tarball Copy snap list to the final report tarball Copy VM name (or none) to the final report tarball Copy current linux distribution to the final report tarball Copy network configuration to the final report tarball Inspecting kubernetes cluster Inspect kubernetes cluster Inspecting dqlite Inspect dqlite cp: cannot stat '/var/snap/microk8s/6673/var/kubernetes/backend/localnode.yaml': No such file or directory

Building the report tarball Report tarball is at /var/snap/microk8s/6673/inspection-report-20240623_091714.tar.gz ubuntu@ip-172-31-37-61:~$ cd /var/snap/microk8s/6673/ ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$ pwd /var/snap/microk8s/6673 ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$ uname -t uname: invalid option -- 't' Try 'uname --help' for more information. ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$ uname -r 6.5.0-1017-aws ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$

and pls find my here microk8s.inspect tar file as well as i generated from secondary nodes. inspection-report.zip

This is from primary node :

ubuntu@ip-172-31-44-251:~$ microk8s kubectl get nodes NAME STATUS ROLES AGE VERSION ip-172-31-44-251 Ready 7m53s v1.26.15 ubuntu@ip-172-31-44-251:~$ microk8s.status microk8s is running high-availability: no datastore master nodes: 172.31.44.251:19001 datastore standby nodes: none addons: enabled: dashboard # (core) The Kubernetes dashboard dns # (core) CoreDNS ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access metrics-server # (core) K8s Metrics Server for API access to service metrics storage # (core) Alias to hostpath-storage add-on, deprecated disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository gpu # (core) Automatic enablement of Nvidia CUDA host-access # (core) Allow Pods connecting to Host services smoothly kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000 ubuntu@ip-172-31-44-251:~$ uname -R uname: invalid option -- 'R' Try 'uname --help' for more information. ubuntu@ip-172-31-44-251:~$ uname -r 6.5.0-1017-aws ubuntu@ip-172-31-44-251:~$

Please help to suggest or advise what is the problem here and why I am unable to join the nodes to form a cluster ?.

@ktsakalozos can you pls help me on this issue.