microk8s
microk8s copied to clipboard
Published
20 hours ago •
canonical
canonical
pod cycle restart,Pod sandbox changed, it will be killed and re-created.
Summary
I installed MicroK8s v1.30.4 revision 7167 on Ubuntu 24.04. Using microK8s status to check the status, MicroK8s is running, but when checking the status of pods, only the Calico node is running, and all other pods are ContainerCreating The ContainerCreating status has been going on for a long time. I have checked the detailed logs and the error reason is: Pod sandbox changed, it will be killed and re created. I used https://github.com/canonical/microk8s/issues/1775 The method in the system involves stopping snap.microk8s.daemon-apiserver picker to stop the apiserver picker service, but the pod still restarts repeatedly. What should I do? thank you.
System:
Ubuntu 24.04
MicroK8s Version:
1.30.4
What Should Happen Instead?
All pods are in running state
Reproduction Steps
- sudo snap install microk8s --classic
- microk8s status --wait-ready
- microk8s enable dashboard
- microk8s enable dns
- microk8s kubectl get all --all-namespaces
- microk8s kubectl get pods --all-namespaces
Detail
> microk8s kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-796fb75cc-8h6m2 0/1 ContainerCreating 0 17h
kube-system calico-node-cf48j 1/1 Running 1 (16h ago) 17h
kube-system coredns-5986966c54-65tdp 0/1 ContainerCreating 0 17h
kube-system dashboard-metrics-scraper-795895d745-72jrs 0/1 ContainerCreating 0 17h
kube-system hostpath-provisioner-7c8bdf94b8-lgszw 0/1 ContainerCreating 0 17h
kube-system kubernetes-dashboard-6796797fb5-bdrsj 0/1 ContainerCreating 0 17h
kube-system metrics-server-7cff7889bd-hx2q6 0/1 ContainerCreating 0 17h
> microk8s version
MicroK8s v1.30.4 revision 7167
> microk8s kubectl describe pods kubernetes-dashboard-6796797fb5-bdrsj -n kube-system
Name: kubernetes-dashboard-6796797fb5-bdrsj
Namespace: kube-system
Priority: 0
Service Account: kubernetes-dashboard
Node: xxxx-pc/192.168.254.131
Start Time: Tue, 03 Sep 2024 22:17:25 +0800
Labels: k8s-app=kubernetes-dashboard
pod-template-hash=6796797fb5
Annotations: <none>
Status: Pending
SeccompProfile: RuntimeDefault
IP:
IPs: <none>
Controlled By: ReplicaSet/kubernetes-dashboard-6796797fb5
Containers:
kubernetes-dashboard:
Container ID:
Image: kubernetesui/dashboard:v2.7.0
Image ID:
Port: 8443/TCP
Host Port: 0/TCP
Args:
--auto-generate-certificates
--namespace=kube-system
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/certs from kubernetes-dashboard-certs (rw)
/tmp from tmp-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2gs6r (ro)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kubernetes-dashboard-certs:
Type: Secret (a volume populated by a Secret)
SecretName: kubernetes-dashboard-certs
Optional: false
tmp-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-2gs6r:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SandboxChanged 63s (x224 over 16h) kubelet Pod sandbox changed, it will be killed and re-created.
> microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
FAIL: Service snap.microk8s.daemon-apiserver-kicker is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-apiserver-kicker
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/7167/var/kubernetes/backend/localnode.yaml': No such file or directory
Building the report tarball
Report tarball is at /var/snap/microk8s/7167/inspection-report-20240904_155434.tar.gz
Hi @pruidong could you please attach the tarball created by the inspection script so we can see the logs of the cluster? Thank you.
Hi @ktsakalozos, I get same symptoms from time to time for now almost one year on microk8s on Mac (From version 1.28 with Multipass 1.13 to 1.31 with multipass 1.14.0).
My microk8s version: MicroK8s v1.31.1 revision 7244. The Microk8s was bootstrapped less than a week ago (On Saturday 27th).
Usually it happens after a few days. I can observe as well errors mentioning networking and calico when it happens (Because of an expired token if I am correct):
● snap.microk8s.daemon-kubelite.service - Service for snap application microk8s.daemon-kubelite
Loaded: loaded (/etc/systemd/system/snap.microk8s.daemon-kubelite.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2024-10-27 23:37:04 CET; 2 days ago
Main PID: 278011 (kubelite)
Tasks: 21 (limit: 19071)
Memory: 747.4M
CPU: 2h 12min 10.103s
CGroup: /system.slice/snap.microk8s.daemon-kubelite.service
└─278011 /snap/microk8s/7244/kubelite --scheduler-args-file=/var/snap/microk8s/7244/args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/7244/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/7244/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/7244/args/kubelet --apiserver-args-file=/var/snap/microk8s/7244/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/7244/credentials/client.config --start-control-plane=true
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092190 278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092252 278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"}
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092282 278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092300 278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-cnpg-1-initdb-wn8pn" podUID="2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a"
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.102166 278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104639 278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104671 278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"}
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104695 278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104708 278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-init-kc5tk" podUID="9d6c80d6-90bf-4f33-a809-0725de373871"
Oct 30 08:20:52 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:52.914986 278011 horizontal.go:275] "Unhandled Error" err="failed to compute desired number of replicas based on listed metrics for Deployment/socrate-local/socrate-saas: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"
It seems to relate to an "invalid bearer token, service account token has expired" that I can observe in kubelite logs:
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072446 278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966"
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072489 278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966"}
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072526 278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"0aa0609f-1413-4115-94ae-b7e1e7adc84a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072545 278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"0aa0609f-1413-4115-94ae-b7e1e7adc84a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="kube-system/hostpath-provisioner-microk8s-vm-l6x2c" podUID="0aa0609f-1413-4115-94ae-b7e1e7adc84a"
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.071560 278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073693 278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073760 278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"}
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073786 278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073801 278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-cnpg-1-initdb-wn8pn" podUID="2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.073937 278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076637 278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076661 278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"}
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076687 278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076703 278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-init-kc5tk" podUID="9d6c80d6-90bf-4f33-a809-0725de373871"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.077419 278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
When I get this issue and that I execute k describe pods ... command I get:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SandboxChanged 42s (x232 over 50m) kubelet Pod sandbox changed, it will be killed and re-created.
daemon-container.d also has interesting logs. Looking at timestamps, you can observe 3 different periods of time where usual and then abnormal things happen (expirer bearer token):
Oct 29 17:43:06 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T17:43:06.594391846+01:00" level=info msg="RemoveContainer for \"7db2460021bb519fa6718499fe137694b60e004b1ca480a064b34810ef647651\""
Oct 29 17:43:06 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T17:43:06.602904388+01:00" level=info msg="RemoveContainer for \"7db2460021bb519fa6718499fe137694b60e004b1ca480a064b34810ef647651\" returns successfully"
Oct 29 18:49:39 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:39.682916298+01:00" level=info msg="StopContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" with timeout 30 (s)"
Oct 29 18:49:39 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:39.683873848+01:00" level=info msg="Stop container \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" with signal terminated"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.030455819+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7896dbf49-2rtwd,Uid:3f29a488-1d2e-4018-82be-da72c5f6e3a9,Namespace:kube-system,Attempt:0,}"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.149 [INFO][2554382] plugin.go 327: Calico CNI found existing endpoint: &{{WorkloadEndpoint projectcalico.org/v3} {microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0 coredns-7896dbf49- kube-system 3f29a488-1d2e-4018-82be-da72c5f6e3a9 333941 0 2024-10-29 18:49:39 +0100 CET <nil> <nil> map[k8s-app:kube-dns pod-template-hash:7896dbf49 projectcalico.org/namespace:kube-system projectcalico.org/orchestrator:k8s projectcalico.org/serviceaccount:coredns] map[] [] [] []} {k8s microk8s-vm coredns-7896dbf49-2rtwd eth0 coredns [] [] [kns.kube-system ksa.kube-system.coredns] cali40c65f0aba7 [{dns UDP 53 0 } {dns-tcp TCP 53 0 } {metrics TCP 9153 0 }] []}} ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.150 [INFO][2554382] k8s.go 74: Extracted identifiers for CmdAddK8s ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.187 [INFO][2554397] ipam_plugin.go 229: Calico CNI IPAM request count IPv4=1 IPv6=0 ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" HandleID="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Workload="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.193 [INFO][2554397] ipam_plugin.go 269: Auto assigning IP ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" HandleID="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Workload="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0x4000695420), Attrs:map[string]string{"namespace":"kube-system", "node":"microk8s-vm", "pod":"coredns-7896dbf49-2rtwd", "timestamp":"2024-10-29 17:49:40.187354331 +0000 UTC"}, Hostname:"microk8s-vm", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}, MaxBlocksPerHost:0, HostReservedAttrIPv4s:(*ipam.HostReservedAttr)(nil), HostReservedAttrIPv6s:(*ipam.HostReservedAttr)(nil), IntendedUse:"Workload"}
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40+01:00" level=info msg="About to acquire host-wide IPAM lock." source="ipam_plugin.go:357"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40+01:00" level=info msg="Acquired host-wide IPAM lock." source="ipam_plugin.go:372"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.194 [INFO][2554397] ipam.go 107: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'microk8s-vm'
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.196 [INFO][2554397] ipam.go 660: Looking up existing affinities for host handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.199 [INFO][2554397] ipam.go 372: Looking up existing affinities for host host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.203 [INFO][2554397] ipam.go 489: Trying affinity for 10.1.254.64/26 host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.204 [INFO][2554397] ipam.go 155: Attempting to load block cidr=10.1.254.64/26 host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.207 [INFO][2554397] ipam.go 232: Affinity is confirmed and block has been loaded cidr=10.1.254.64/26 host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.207 [INFO][2554397] ipam.go 1180: Attempting to assign 1 addresses from block block=10.1.254.64/26 handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.210 [INFO][2554397] ipam.go 1682: Creating new handle: k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.219 [INFO][2554397] ipam.go 1203: Writing block in order to claim IPs block=10.1.254.64/26 handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.230 [INFO][2554397] ipam.go 1216: Successfully claimed IPs: [10.1.254.126/26] block=10.1.254.64/26 handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.230 [INFO][2554397] ipam.go 847: Auto-assigned 1 out of 1 IPv4s: [10.1.254.126/26] handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40+01:00" level=info msg="Released host-wide IPAM lock." source="ipam_plugin.go:378"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.230 [INFO][2554397] ipam_plugin.go 287: Calico CNI IPAM assigned addresses IPv4=[10.1.254.126/26] IPv6=[] ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" HandleID="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Workload="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.232 [INFO][2554382] k8s.go 383: Populated endpoint ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0", GenerateName:"coredns-7896dbf49-", Namespace:"kube-system", SelfLink:"", UID:"3f29a488-1d2e-4018-82be-da72c5f6e3a9", ResourceVersion:"333941", Generation:0, CreationTimestamp:time.Date(2024, time.October, 29, 18, 49, 39, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-dns", "pod-template-hash":"7896dbf49", "projectcalico.org/namespace":"kube-system", "projectcalico.org/orchestrator":"k8s", "projectcalico.org/serviceaccount":"coredns"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"microk8s-vm", ContainerID:"", Pod:"coredns-7896dbf49-2rtwd", Endpoint:"eth0", ServiceAccountName:"coredns", IPNetworks:[]string{"10.1.254.126/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.kube-system", "ksa.kube-system.coredns"}, InterfaceName:"cali40c65f0aba7", MAC:"", Ports:[]v3.WorkloadEndpointPort{v3.WorkloadEndpointPort{Name:"dns", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"UDP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"dns-tcp", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"metrics", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x23c1, HostPort:0x0, HostIP:""}}, AllowSpoofedSourcePrefixes:[]string(nil)}}
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.232 [INFO][2554382] k8s.go 384: Calico CNI using IPs: [10.1.254.126/32] ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.232 [INFO][2554382] dataplane_linux.go 68: Setting the host side veth name to cali40c65f0aba7 ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.234 [INFO][2554382] dataplane_linux.go 473: Disabling IPv4 forwarding ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.278 [INFO][2554382] k8s.go 411: Added Mac, interface name, and active container ID to endpoint ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0", GenerateName:"coredns-7896dbf49-", Namespace:"kube-system", SelfLink:"", UID:"3f29a488-1d2e-4018-82be-da72c5f6e3a9", ResourceVersion:"333941", Generation:0, CreationTimestamp:time.Date(2024, time.October, 29, 18, 49, 39, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-dns", "pod-template-hash":"7896dbf49", "projectcalico.org/namespace":"kube-system", "projectcalico.org/orchestrator":"k8s", "projectcalico.org/serviceaccount":"coredns"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"microk8s-vm", ContainerID:"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe", Pod:"coredns-7896dbf49-2rtwd", Endpoint:"eth0", ServiceAccountName:"coredns", IPNetworks:[]string{"10.1.254.126/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.kube-system", "ksa.kube-system.coredns"}, InterfaceName:"cali40c65f0aba7", MAC:"56:82:52:7a:b1:60", Ports:[]v3.WorkloadEndpointPort{v3.WorkloadEndpointPort{Name:"dns", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"UDP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"dns-tcp", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"metrics", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x23c1, HostPort:0x0, HostIP:""}}, AllowSpoofedSourcePrefixes:[]string(nil)}}
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.287 [INFO][2554382] k8s.go 489: Wrote updated endpoint to datastore ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.310464553+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.310634054+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.310646221+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.313131701+01:00" level=info msg="starting signal loop" namespace=k8s.io path=/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe pid=2554432 runtime=io.containerd.runc.v2
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.349406220+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7896dbf49-2rtwd,Uid:3f29a488-1d2e-4018-82be-da72c5f6e3a9,Namespace:kube-system,Attempt:0,} returns sandbox id \"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe\""
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.351855699+01:00" level=info msg="CreateContainer within sandbox \"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe\" for container &ContainerMetadata{Name:coredns,Attempt:0,}"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.367374541+01:00" level=info msg="CreateContainer within sandbox \"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe\" for &ContainerMetadata{Name:coredns,Attempt:0,} returns container id \"8f83898d31719f4ef8dd6b96eee44a783275b4c7f4a6052275ec87b5235a092a\""
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.367810045+01:00" level=info msg="StartContainer for \"8f83898d31719f4ef8dd6b96eee44a783275b4c7f4a6052275ec87b5235a092a\""
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.401905754+01:00" level=info msg="StartContainer for \"8f83898d31719f4ef8dd6b96eee44a783275b4c7f4a6052275ec87b5235a092a\" returns successfully"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.786493665+01:00" level=info msg="shim disconnected" id=7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.786535498+01:00" level=warning msg="cleaning up after shim disconnected" id=7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0 namespace=k8s.io
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.786543207+01:00" level=info msg="cleaning up dead shim"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.791813960+01:00" level=warning msg="cleanup warnings time=\"2024-10-29T18:49:44+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2554683 runtime=io.containerd.runc.v2\n"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.796758294+01:00" level=info msg="StopContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" returns successfully"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.797232798+01:00" level=info msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\""
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.797308382+01:00" level=info msg="Container to stop \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.826756468+01:00" level=info msg="shim disconnected" id=3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.826956553+01:00" level=warning msg="cleaning up after shim disconnected" id=3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d namespace=k8s.io
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.826966261+01:00" level=info msg="cleaning up dead shim"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.832047472+01:00" level=warning msg="cleanup warnings time=\"2024-10-29T18:49:44+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2554714 runtime=io.containerd.runc.v2\n"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.872 [INFO][2554742] k8s.go 576: Cleaning up netns ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.872 [INFO][2554742] dataplane_linux.go 524: Deleting workload's device in netns. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" iface="eth0" netns="/var/run/netns/cni-eeebfd87-37c4-6db9-dad8-b61ec4390400"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.872 [INFO][2554742] dataplane_linux.go 535: Entered netns, deleting veth. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" iface="eth0" netns="/var/run/netns/cni-eeebfd87-37c4-6db9-dad8-b61ec4390400"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.946 [INFO][2554742] dataplane_linux.go 569: Deleted device in netns. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" after=73.601047ms iface="eth0" netns="/var/run/netns/cni-eeebfd87-37c4-6db9-dad8-b61ec4390400"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.946 [INFO][2554742] k8s.go 583: Releasing IP address(es) ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.946 [INFO][2554742] utils.go 195: Calico CNI releasing IP address ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.965 [INFO][2554761] ipam_plugin.go 416: Releasing address using handleID ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" HandleID="k8s-pod-network.3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" Workload="microk8s--vm-k8s-coredns--7896dbf49--l5bpm-eth0"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44+01:00" level=info msg="About to acquire host-wide IPAM lock." source="ipam_plugin.go:357"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44+01:00" level=info msg="Acquired host-wide IPAM lock." source="ipam_plugin.go:372"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.998 [INFO][2554761] ipam_plugin.go 435: Released address using handleID ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" HandleID="k8s-pod-network.3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" Workload="microk8s--vm-k8s-coredns--7896dbf49--l5bpm-eth0"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.998 [INFO][2554761] ipam_plugin.go 444: Releasing address using workloadID ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" HandleID="k8s-pod-network.3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" Workload="microk8s--vm-k8s-coredns--7896dbf49--l5bpm-eth0"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45+01:00" level=info msg="Released host-wide IPAM lock." source="ipam_plugin.go:378"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:45.001 [INFO][2554742] k8s.go 589: Teardown processing complete. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.002903477+01:00" level=info msg="TearDown network for sandbox \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\" successfully"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.002927686+01:00" level=info msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\" returns successfully"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.748839160+01:00" level=info msg="RemoveContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\""
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.754594751+01:00" level=info msg="RemoveContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" returns successfully"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.755135381+01:00" level=error msg="ContainerStatus for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\": not found"
Oct 30 07:40:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:52.972005685+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-mq89l,Uid:a5ca97f8-e262-44c7-a5e1-677a73531613,Namespace:kube-system,Attempt:0,}"
Oct 30 07:40:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:52.978256196+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-l6x2c,Uid:0aa0609f-1413-4115-94ae-b7e1e7adc84a,Namespace:kube-system,Attempt:0,}"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.061891021+01:00" level=error msg="Failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\"" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.062489110+01:00" level=error msg="Failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\"" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063047364+01:00" level=error msg="encountered an error cleaning up failed sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063107573+01:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-mq89l,Uid:a5ca97f8-e262-44c7-a5e1-677a73531613,Namespace:kube-system,Attempt:0,} failed, error" error="failed to setup network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063361450+01:00" level=error msg="encountered an error cleaning up failed sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063388742+01:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-l6x2c,Uid:0aa0609f-1413-4115-94ae-b7e1e7adc84a,Namespace:kube-system,Attempt:0,} failed, error" error="failed to setup network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.301356443+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.301518653+01:00" level=info msg="Ensure that sandbox 1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966 in task-service has been cleanup successfully"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.302520536+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.302665579+01:00" level=info msg="Ensure that sandbox 6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35 in task-service has been cleanup successfully"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.333379253+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.336035984+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:04 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:04.045217580+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:04 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:04.084340824+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:07 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:07.796526994+01:00" level=info msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\""
Oct 30 07:41:07 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:07.826715372+01:00" level=error msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\" failed" error="failed to destroy network for sandbox \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:08 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:08.048635688+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:08 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:08.076329212+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:17 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:17.042324007+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:17 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:17.082576210+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:22 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:22.050095281+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:22 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:22.078859166+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:29 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:29.043485545+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:29 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:29.073175607+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:35 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:35.045591333+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:35 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:35.077397566+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:42 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:42.044458439+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:42 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:42.071405788+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:48 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:48.043846844+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:48 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:48.098057521+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.909196319+01:00" level=info msg="StopContainer for \"93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e\" with timeout 30 (s)"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.911117520+01:00" level=info msg="Stop container \"93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e\" with signal terminated"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.977787809+01:00" level=info msg="shim disconnected" id=93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.977854435+01:00" level=warning msg="cleaning up after shim disconnected" id=93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e namespace=k8s.io
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.977869602+01:00" level=info msg="cleaning up dead shim"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.984172008+01:00" level=warning msg="cleanup warnings time=\"2024-10-30T07:41:52+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2558950 runtime=io.containerd.runc.v2\n"
Does it help ?
One additional info: I finally decided to reboot the VM (multipass shell, then sudo reboot / No worry, it is a dev env), and it seems to fix my problem, I can start pods again after the reboot. I assume that start phase of k8s parts renew the bearer token that causes problems.
Still, I don't understand to which token it relates. If anyone can give me clarity would be awesome.
I have the same problem, @pruidong have you find a solution?
@ktsakalozos here are my inspection report: inspection-report-20241115_150550.tar.gz
Reboot the machine does nothing with this matter for me.
Ok, found this https://stackoverflow.com/questions/66516049/microk8s-pods-are-restarting-frequently-on-my-raspberry-pi-ubuntu Disabling ipv6 worked for me. Hope it helps
I still have the same issue with 1.31.2 (update from 1.25 step by step) - any other resolution on that?
Same here. It has happened to me twice after multipass suspend and then shell after a few days: I am no longer able to use microk8s.
Rebooting the VM does work, but it's inconvenient.
