for-win icon indicating copy to clipboard operation
for-win copied to clipboard

Published 20 hours ago • verified publisher icon docker

Kubelet failed to start

Open stevefan1999-personal opened this issue 1 year ago • 11 comments

Description

Kubernetes cannot be started with Docker Desktop due to this error: failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing controllers: cpu

Reproduce

  1. Enable Kubernetes
  2. Restart Docker Desktop

Expected behavior

No response

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:48 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.25.2 (129061)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 62
  Running: 0
  Paused: 0
  Stopped: 62
 Images: 52
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.31GiB
 Name: Steve-Work
 ID: 6fd128fb-f844-42f9-9925-a4278a03b9c5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

95360111-1900-4527-B0E1-4DC3738A7031/20231204111139

Additional Info

kubelet.log:

-------------------------------------------------------------------------------->8
[2023-12-04T10:57:35.715956481Z][kubelet][I] Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[2023-12-04T10:57:35.722411601Z][kubelet][I] I1204 10:57:35.722184     998 server.go:467] "Kubelet version" kubeletVersion="v1.28.2"
[2023-12-04T10:57:35.722442859Z][kubelet][I] I1204 10:57:35.722244     998 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
[2023-12-04T10:57:35.730163587Z][kubelet][I] I1204 10:57:35.729947     998 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt"
[2023-12-04T10:57:35.781493007Z][kubelet][I] W1204 10:57:35.781321     998 sysinfo.go:203] Nodes topology is not available, providing CPU topology
[2023-12-04T10:57:35.792733291Z][kubelet][I] E1204 10:57:35.792580     998 run.go:74] "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing controllers: cpu"
[2023-12-04T10:57:35.837653778Z][kubelet][I] EOF

stevefan1999-personal avatar Dec 04 '23 11:12 stevefan1999-personal

I've observed something similar but with a different reason for invalid configuration:

[2024-02-04T17:53:43.226119600Z][kubelet][I] Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[2024-02-04T17:53:43.236734900Z][kubelet][I] I0204 17:53:43.236455     893 server.go:487] "Kubelet version" kubeletVersion="v1.29.1"
[2024-02-04T17:53:43.236768100Z][kubelet][I] I0204 17:53:43.236560     893 server.go:489] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
[2024-02-04T17:53:43.253521200Z][kubelet][I] I0204 17:53:43.253275     893 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt"
[2024-02-04T17:53:43.297379300Z][kubelet][I] E0204 17:53:43.297203     893 run.go:74] "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"
[2024-02-04T17:53:43.333398800Z][kubelet][I] EOF

This seems to only happen when Docker Desktop starts for the first time after powering on my host machine. If I restart Docker Desktop via the taskbar after that, kubernetes comes up fine.

robjackstewart avatar Feb 04 '24 18:02 robjackstewart

same issue as @robjackstewart

[2024-02-21T21:35:04.507142732Z][kubelet][I] W0221 21:35:04.506792     798 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.Service: Get "https://kubernetes.docker.internal:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-02-21T21:35:04.507184337Z][kubelet][I] E0221 21:35:04.506926     798 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://kubernetes.docker.internal:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-02-21T21:35:04.600504144Z][kubelet][I] I0221 21:35:04.600347     798 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc0d1d57e72179e977a31de89064f90ec2c3f0a506d603a9938120529a40e14"
[2024-02-21T21:35:04.616845804Z][kubelet][I] I0221 21:35:04.616490     798 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56d2d75ea4b47083dd4e265962986d3b4390ec3a4cb5f191e185657dc3996afd"
[2024-02-21T21:35:04.624526503Z][kubelet][I] I0221 21:35:04.624355     798 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ca9e2fee04cc66f3918c20375bad53bba006e5896e3c7483c804d94b62aee9"
[2024-02-21T21:35:04.632029535Z][kubelet][I] I0221 21:35:04.631768     798 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4053f05a0f2cfb6688e6c247a2d71c3ffefc0b8ba62f145a9eb98926266d325"
[2024-02-21T21:35:04.869773719Z][kubelet][I] E0221 21:35:04.869506     798 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused" interval="1.6s"
[2024-02-21T21:35:04.889228552Z][kubelet][I] W0221 21:35:04.889067     798 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.Node: Get "https://kubernetes.docker.internal:6443/api/v1/nodes?fieldSelector=metadata.name%3Ddocker-desktop&limit=500&resourceVersion=0": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-02-21T21:35:04.889244366Z][kubelet][I] E0221 21:35:04.889124     798 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://kubernetes.docker.internal:6443/api/v1/nodes?fieldSelector=metadata.name%3Ddocker-desktop&limit=500&resourceVersion=0": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-02-21T21:35:04.961387108Z][kubelet][I] W0221 21:35:04.961222     798 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.CSIDriver: Get "https://kubernetes.docker.internal:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-02-21T21:35:04.961430915Z][kubelet][I] E0221 21:35:04.961280     798 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://kubernetes.docker.internal:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-02-21T21:35:04.984329298Z][kubelet][I] I0221 21:35:04.984209     798 kubelet_node_status.go:73] "Attempting to register node" node="docker-desktop"
[2024-02-21T21:35:04.986152763Z][kubelet][I] E0221 21:35:04.986053     798 kubelet_node_status.go:96] "Unable to register node with API server" err="Post \"https://kubernetes.docker.internal:6443/api/v1/nodes\": dial tcp 192.168.65.3:6443: connect: connection refused" node="docker-desktop"
[2024-02-21T21:35:06.593095768Z][kubelet][I] I0221 21:35:06.592671     798 kubelet_node_status.go:73] "Attempting to register node" node="docker-desktop"
[2024-02-21T21:35:07.675735801Z][kubelet][I] I0221 21:35:07.675541     798 kubelet_node_status.go:112] "Node was previously registered" node="docker-desktop"
[2024-02-21T21:35:07.675833639Z][kubelet][I] I0221 21:35:07.675647     798 kubelet_node_status.go:76] "Successfully registered node" node="docker-desktop"
[2024-02-21T21:35:07.704294602Z][kubelet][I] E0221 21:35:07.704084     798 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"docker-desktop\" not found"
[2024-02-21T21:35:07.805301723Z][kubelet][I] E0221 21:35:07.804894     798 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"docker-desktop\" not found"
[2024-02-21T21:35:08.450279186Z][kubelet][I] I0221 21:35:08.449256     798 apiserver.go:52] "Watching apiserver"
[2024-02-21T21:35:08.463446097Z][kubelet][I] I0221 21:35:08.463156     798 topology_manager.go:215] "Topology Admit Handler" podUID="54967b7b-f88b-44c8-9a25-41506ac80ec7" podNamespace="kube-system" podName="kube-proxy-qlwdq"
[2024-02-21T21:35:08.463902168Z][kubelet][I] I0221 21:35:08.463638     798 topology_manager.go:215] "Topology Admit Handler" podUID="48ec42f8-bc15-4a68-8e9a-a9ae9afff81b" podNamespace="kube-system" podName="coredns-76f75df574-q4tc5"
[2024-02-21T21:35:08.463946499Z][kubelet][I] I0221 21:35:08.463767     798 topology_manager.go:215] "Topology Admit Handler" podUID="53ce24d1-1d10-4cf7-8c4e-8161d7473622" podNamespace="kube-system" podName="coredns-76f75df574-q5f54"
[2024-02-21T21:35:08.467114467Z][kubelet][I] I0221 21:35:08.465215     798 topology_manager.go:215] "Topology Admit Handler" podUID="caf38982-7e89-4aa9-b877-10dbb0a77b86" podNamespace="kube-system" podName="storage-provisioner"
[2024-02-21T21:35:08.467155109Z][kubelet][I] I0221 21:35:08.465293     798 topology_manager.go:215] "Topology Admit Handler" podUID="d7c2fb1f-fb99-4448-aeb7-989ce653cdf4" podNamespace="kube-system" podName="vpnkit-controller"
[2024-02-21T21:35:08.482727299Z][kubelet][I] E0221 21:35:08.482150     798 kubelet.go:1921] "Failed creating a mirror pod for" err="pods \"etcd-docker-desktop\" already exists" pod="kube-system/etcd-docker-desktop"
[2024-02-21T21:35:08.553341018Z][kubelet][I] I0221 21:35:08.552828     798 desired_state_of_world_populator.go:159] "Finished populating initial desired state of world"
[2024-02-21T21:35:08.576558683Z][kubelet][I] I0221 21:35:08.576433     798 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvs\" (UniqueName: \"kubernetes.io/host-path/caf38982-7e89-4aa9-b877-10dbb0a77b86-pvs\") pod \"storage-provisioner\" (UID: \"caf38982-7e89-4aa9-b877-10dbb0a77b86\") " pod="kube-system/storage-provisioner"
[2024-02-21T21:35:08.576593724Z][kubelet][I] I0221 21:35:08.576475     798 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/54967b7b-f88b-44c8-9a25-41506ac80ec7-xtables-lock\") pod \"kube-proxy-qlwdq\" (UID: \"54967b7b-f88b-44c8-9a25-41506ac80ec7\") " pod="kube-system/kube-proxy-qlwdq"
[2024-02-21T21:35:08.576603301Z][kubelet][I] I0221 21:35:08.576490     798 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54967b7b-f88b-44c8-9a25-41506ac80ec7-lib-modules\") pod \"kube-proxy-qlwdq\" (UID: \"54967b7b-f88b-44c8-9a25-41506ac80ec7\") " pod="kube-system/kube-proxy-qlwdq"
[2024-02-21T21:35:08.576603916Z][kubelet][I] I0221 21:35:08.576504     798 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"api\" (UniqueName: \"kubernetes.io/host-path/d7c2fb1f-fb99-4448-aeb7-989ce653cdf4-api\") pod \"vpnkit-controller\" (UID: \"d7c2fb1f-fb99-4448-aeb7-989ce653cdf4\") " pod="kube-system/vpnkit-controller"
[2024-02-21T21:35:11.867804419Z][kubelet][I] I0221 21:35:11.867611     798 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
[2024-02-21T21:35:15.006816433Z][kubelet][I] I0221 21:35:15.006581     798 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
-------------------------------------------------------------------------------->8
[2024-02-21T21:38:06.125138391Z][kubelet][I] Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[2024-02-21T21:38:06.130638108Z][kubelet][I] I0221 21:38:06.130530     834 server.go:487] "Kubelet version" kubeletVersion="v1.29.1"
[2024-02-21T21:38:06.130657471Z][kubelet][I] I0221 21:38:06.130579     834 server.go:489] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
[2024-02-21T21:38:06.139407530Z][kubelet][I] I0221 21:38:06.139311     834 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt"
[2024-02-21T21:38:06.163438390Z][kubelet][I] E0221 21:38:06.163290     834 run.go:74] "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"
[2024-02-21T21:38:06.209086288Z][kubelet][I] EOF

rishabh911996 avatar Feb 21 '24 21:02 rishabh911996

Are there any insights on this? We are experiencing the exact same behavior.

[2024-03-07T10:32:41.617552843Z][kubelet][I] E0307 10:32:41.617327     768 kubelet.go:1921] "Failed creating a mirror pod for" err="pods \"kube-apiserver-docker-desktop\" already exists" pod="kube-system/kube-apiserver-docker-desktop"
[2024-03-07T10:32:42.030938439Z][kubelet][I] I0307 10:32:42.030788     768 apiserver.go:52] "Watching apiserver"
[2024-03-07T10:32:42.037978037Z][kubelet][I] I0307 10:32:42.037723     768 topology_manager.go:215] "Topology Admit Handler" podUID="ba268470-6b27-4727-b17c-54a24998932c" podNamespace="kube-system" podName="coredns-76f75df574-r9wrr"
[2024-03-07T10:32:42.038155946Z][kubelet][I] I0307 10:32:42.038071     768 topology_manager.go:215] "Topology Admit Handler" podUID="57bb2ad0-bbbb-4500-b054-977b272fc7fb" podNamespace="kube-system" podName="kube-proxy-vhsfk"
[2024-03-07T10:32:42.040020554Z][kubelet][I] I0307 10:32:42.038345     768 topology_manager.go:215] "Topology Admit Handler" podUID="87a15f66-4e3d-48a5-8a84-cbc952e951b4" podNamespace="kube-system" podName="coredns-76f75df574-5twzh"
[2024-03-07T10:32:42.040035532Z][kubelet][I] I0307 10:32:42.038496     768 topology_manager.go:215] "Topology Admit Handler" podUID="e2119797-7284-4c74-b192-772f1b8cc7f4" podNamespace="kube-system" podName="vpnkit-controller"
[2024-03-07T10:32:42.040049013Z][kubelet][I] I0307 10:32:42.038580     768 topology_manager.go:215] "Topology Admit Handler" podUID="6622b371-6369-435e-a35f-87e20510f7d0" podNamespace="kube-system" podName="storage-provisioner"
[2024-03-07T10:32:42.041238584Z][kubelet][I] I0307 10:32:42.041084     768 desired_state_of_world_populator.go:159] "Finished populating initial desired state of world"
[2024-03-07T10:32:42.129935028Z][kubelet][I] I0307 10:32:42.129744     768 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57bb2ad0-bbbb-4500-b054-977b272fc7fb-lib-modules\") pod \"kube-proxy-vhsfk\" (UID: \"57bb2ad0-bbbb-4500-b054-977b272fc7fb\") " pod="kube-system/kube-proxy-vhsfk"
[2024-03-07T10:32:42.130032256Z][kubelet][I] I0307 10:32:42.129830     768 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"api\" (UniqueName: \"kubernetes.io/host-path/e2119797-7284-4c74-b192-772f1b8cc7f4-api\") pod \"vpnkit-controller\" (UID: \"e2119797-7284-4c74-b192-772f1b8cc7f4\") " pod="kube-system/vpnkit-controller"
[2024-03-07T10:32:42.130042161Z][kubelet][I] I0307 10:32:42.129956     768 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvs\" (UniqueName: \"kubernetes.io/host-path/6622b371-6369-435e-a35f-87e20510f7d0-pvs\") pod \"storage-provisioner\" (UID: \"6622b371-6369-435e-a35f-87e20510f7d0\") " pod="kube-system/storage-provisioner"
[2024-03-07T10:32:42.130738433Z][kubelet][I] I0307 10:32:42.130269     768 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/57bb2ad0-bbbb-4500-b054-977b272fc7fb-xtables-lock\") pod \"kube-proxy-vhsfk\" (UID: \"57bb2ad0-bbbb-4500-b054-977b272fc7fb\") " pod="kube-system/kube-proxy-vhsfk"
[2024-03-07T10:32:43.946464070Z][kubelet][I] I0307 10:32:43.946276     768 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819a79ba04950632cc33da4e8f8f47131b64f6a8352d5b112f900a9e4753c777"
[2024-03-07T10:32:45.093034593Z][kubelet][I] I0307 10:32:45.092910     768 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4465ae924b70c44bee87af77f7a069ef0d8de4b26e6732105f4e38264f05cf0b"
[2024-03-07T10:32:45.346966981Z][kubelet][I] I0307 10:32:45.346754     768 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8177d8b8983da526458878aeb9e967a29aef18419b67142238cfb1a878f38f70"
[2024-03-07T10:32:45.362989940Z][kubelet][I] I0307 10:32:45.362758     768 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d2bdbf3a6bfb29cd0e85c546da516eeaa071860a109e935df2b53d8861448e"
[2024-03-07T10:32:47.481985933Z][kubelet][I] I0307 10:32:47.481627     768 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
[2024-03-07T10:32:50.343560152Z][kubelet][I] I0307 10:32:50.343394     768 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
[2024-03-07T10:32:53.361710731Z][kubelet][I] W0307 10:32:53.360039     768 reflector.go:462] vendor/k8s.io/client-go/informers/factory.go:159: watch of *v1.CSIDriver ended with: Internal error occurred: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
[2024-03-07T10:32:53.361741549Z][kubelet][I] W0307 10:32:53.360085     768 reflector.go:462] vendor/k8s.io/client-go/informers/factory.go:159: watch of *v1.RuntimeClass ended with: Internal error occurred: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
[2024-03-07T10:32:54.036205250Z][kubelet][I] E0307 10:32:54.036084     768 event.go:355] "Unable to write event (may retry after sleeping)" err="Post \"https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/events\": dial tcp 192.168.65.3:6443: connect: connection refused" event="&Event{ObjectMeta:{kube-apiserver-docker-desktop.17ba74edfc942501  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-docker-desktop,UID:0ebf02f01020bac6394d8c559802bcc8,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-apiserver},},Reason:Unhealthy,Message:Readiness probe failed: Get \"https://192.168.65.3:6443/readyz\": dial tcp 192.168.65.3:6443: connect: connection refused,Source:EventSource{Component:kubelet,Host:docker-desktop,},FirstTimestamp:2024-03-07 10:32:54.034162945 +0000 UTC m=+20.121864085,LastTimestamp:2024-03-07 10:32:54.034162945 +0000 UTC m=+20.121864085,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:n
[2024-03-07T10:32:59.071218028Z][kubelet][I] W0307 10:32:59.071048     768 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.CSIDriver: Get "https://kubernetes.docker.internal:6443/apis/storage.k8s.io/v1/csidrivers?resourceVersion=461": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-03-07T10:32:59.071237727Z][kubelet][I] E0307 10:32:59.071113     768 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://kubernetes.docker.internal:6443/apis/storage.k8s.io/v1/csidrivers?resourceVersion=461": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-03-07T10:33:00.026797721Z][kubelet][I] W0307 10:33:00.026675     768 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.RuntimeClass: Get "https://kubernetes.docker.internal:6443/apis/node.k8s.io/v1/runtimeclasses?resourceVersion=521": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-03-07T10:33:00.026818185Z][kubelet][I] E0307 10:33:00.026736     768 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://kubernetes.docker.internal:6443/apis/node.k8s.io/v1/runtimeclasses?resourceVersion=521": dial tcp 192.168.65.3:6443: connect: connection refused
[2024-03-07T10:33:01.207009881Z][kubelet][I] E0307 10:33:01.206895     768 controller.go:195] "Failed to update lease" err="Put \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused"
[2024-03-07T10:33:01.208535577Z][kubelet][I] E0307 10:33:01.208407     768 controller.go:195] "Failed to update lease" err="Put \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused"
[2024-03-07T10:33:01.210133246Z][kubelet][I] E0307 10:33:01.210000     768 controller.go:195] "Failed to update lease" err="Put \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused"
[2024-03-07T10:33:01.212753769Z][kubelet][I] E0307 10:33:01.212616     768 controller.go:195] "Failed to update lease" err="Put \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused"
[2024-03-07T10:33:01.214339060Z][kubelet][I] E0307 10:33:01.214238     768 controller.go:195] "Failed to update lease" err="Put \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused"
[2024-03-07T10:33:01.214357504Z][kubelet][I] I0307 10:33:01.214275     768 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
[2024-03-07T10:33:01.215854217Z][kubelet][I] E0307 10:33:01.215738     768 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused" interval="200ms"
[2024-03-07T10:33:01.418878895Z][kubelet][I] E0307 10:33:01.418756     768 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused" interval="400ms"
[2024-03-07T10:33:01.823446507Z][kubelet][I] E0307 10:33:01.823110     768 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused" interval="800ms"
[2024-03-07T10:33:02.626991290Z][kubelet][I] E0307 10:33:02.626778     768 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop?timeout=30s\": dial tcp 192.168.65.3:6443: connect: connection refused" interval="1.6s"
[2024-03-07T10:33:02.974958371Z][kubelet][I] E0307 10:33:02.974827     768 event.go:355] "Unable to write event (may retry after sleeping)" err="Post \"https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/events\": dial tcp 192.168.65.3:6443: connect: connection refused" event="&Event{ObjectMeta:{kube-apiserver-docker-desktop.17ba74edfc942501  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-docker-desktop,UID:0ebf02f01020bac6394d8c559802bcc8,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-apiserver},},Reason:Unhealthy,Message:Readiness probe failed: Get \"https://192.168.65.3:6443/readyz\": dial tcp 192.168.65.3:6443: connect: connection refused,Source:EventSource{Component:kubelet,Host:docker-desktop,},FirstTimestamp:2024-03-07 10:32:54.034162945 +0000 UTC m=+20.121864085,LastTimestamp:2024-03-07 10:32:54.034162945 +0000 UTC m=+20.121864085,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:n
-------------------------------------------------------------------------------->8
[2024-03-07T10:37:45.915734428Z][kubelet][I] Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[2024-03-07T10:37:46.271787784Z][kubelet][I] I0307 10:37:46.271438    1192 server.go:487] "Kubelet version" kubeletVersion="v1.29.1"
[2024-03-07T10:37:46.271891450Z][kubelet][I] I0307 10:37:46.271548    1192 server.go:489] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
[2024-03-07T10:37:46.276365529Z][kubelet][I] I0307 10:37:46.276270    1192 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt"
[2024-03-07T10:37:46.288084986Z][kubelet][I] E0307 10:37:46.287955    1192 run.go:74] "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"
[2024-03-07T10:37:46.343931091Z][kubelet][I] EOF
-------------------------------------------------------------------------------->8
[2024-03-07T11:00:33.083947305Z][kubelet][I] Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[2024-03-07T11:00:33.655015922Z][kubelet][I] I0307 11:00:33.654764     946 server.go:487] "Kubelet version" kubeletVersion="v1.29.1"
[2024-03-07T11:00:33.655040665Z][kubelet][I] I0307 11:00:33.654839     946 server.go:489] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
[2024-03-07T11:00:33.658712949Z][kubelet][I] I0307 11:00:33.658488     946 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt"
[2024-03-07T11:00:33.669642501Z][kubelet][I] E0307 11:00:33.669523     946 run.go:74] "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"
[2024-03-07T11:00:33.715304326Z][kubelet][I] EOF
-------------------------------------------------------------------------------->8
[2024-03-07T11:26:35.815016630Z][kubelet][I] Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
[2024-03-07T11:26:36.118349710Z][kubelet][I] I0307 11:26:36.118062     950 server.go:487] "Kubelet version" kubeletVersion="v1.29.1"
[2024-03-07T11:26:36.118422117Z][kubelet][I] I0307 11:26:36.118129     950 server.go:489] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
[2024-03-07T11:26:36.123663027Z][kubelet][I] I0307 11:26:36.123498     950 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/run/config/pki/ca.crt"
[2024-03-07T11:26:36.136150088Z][kubelet][I] E0307 11:26:36.135751     950 run.go:74] "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"

EvgeniaPatsoni avatar Mar 11 '24 14:03 EvgeniaPatsoni

Thanks for reporting, based on our observation so far, this happens only when using a non-docker distro (e.g. Ubuntu). The workaround for this is to remove those non-docker distros and keep only the "docker-desktop" and "docker-desktop-data" distros.

Example:

  1. In PowerShell, run the below command to set docker-desktop as the default WSL 2 distro: wsl --set-default docker-desktop 2 Reference: https://docs.docker.com/desktop/wsl/#enabling-docker-support-in-wsl-2-distros

  2. In Powershell, run the below command to remove the non-docker WSL distro. (For example, Ubuntu-22.04) wsl --unregister Ubuntu-22.04 Reference: https://learn.microsoft.com/en-us/windows/wsl/faq#how-do-i-uninstall-a-wsl-distribution-

  3. Close Docker Desktop.

  4. Shutdown WSL by running the command below: wsl --shutdown

  5. Open Docker Desktop to monitor the Kubernetes's status, if you still experiencing the same issue, a “Reset Kubernetes Cluster” might be needed after performing the above.

  6. Reboot the machine, and open Docker Desktop to see if you can get a healthy Kubernetes.

voonyee avatar Mar 13 '24 09:03 voonyee

I too see the exact same above behavior with Ubuntu installed. Removing ubuntu did in fact fix the problem, but now I can't use Ubuntu. Works temporarily, but not a long term solution.

jim-olsen avatar Mar 15 '24 19:03 jim-olsen

@voonyee thanks for the workaround. Can you elaborate upon what specifically is causing the problem on the non-docker WSL distro? Is there something that needs to be done on the non-docker WSL distro to be made more compatible with Docker Desktop?

robjackstewart avatar Mar 17 '24 10:03 robjackstewart

Thanks for applying the workaround. From the Diagnostics log provided, the issue is related to the below error and caused the Kubernetes cluster to fail to start: "command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"

Currently, Engineering is investigating this, however, there is no estimated time of fix to share at the moment. We will share when the fix for this is available. Sorry for the inconvenience caused.

voonyee avatar Mar 18 '24 01:03 voonyee

The problem is indeed in multiple VMs installed in WSL environment. When docker+k8s wants to do its stuff, there is no "engine" in other non-prepared environments (such as plain ubuntu or what ever distro user has installed next to docker-desktop wsl instance. The hack with selecting default distro works but it makes life tough for other scenarios because then user must write commands like wsl -d "Ubuntu" etc to get back into his flow. For this situation, it would be feasible to let user decide what distro should be use as a WSL backend... so maybe add some option to JSON/any configuration?

doublemcz avatar Mar 25 '24 05:03 doublemcz

For me the issue occurs as well, when Ubuntu is not the default distro, but in Docker Desktop settings WSL integration is enabled for the Ubuntu distro (Resources > WSL integration > Enable integration with additional distros: Ubuntu). There seems no workaround for this. The last version of Docker Desktop which works fine for me is 4.26.1.

niels-schmitt avatar Mar 27 '24 14:03 niels-schmitt

Here is how I did solve the problem:

Disable Docker Desktop WSL Integration.

  1. Open Docker Desktop GUI
  2. Go to Settings > Resources > WSL Integration
  3. Uncheck Enable integration with my default WSL distro option.
  4. Uncheck any additional distros
  5. Restart Docker Desktop
  6. If it’s hanged,
    1. Kill Docker Desktop, Docker Desktop Extensions and Docker Desktop Background processes
    2. Shutdown WSL with wsl --shutdown --all
  7. Reboot Windows

compadrejunior avatar Mar 31 '24 16:03 compadrejunior

The solution from @compadrejunior worked for a new installation that was failing for my colleague on Win-11 using docker v4.28.0
However it's notable that I don't encounter the same issue running on Win-10 after upgrading to docker v4.28.0

crowne avatar Apr 03 '24 10:04 crowne

I had this same problem.

I could only get Kubernetes to run after doing a full clean/purge data (debug icon in Docker desktop). Once I restarted the computer, Kubernetes would hang on "starting Kubernetes" in the settings panel. I went looking for logs and then found the kubelet log file located in C:\Users\chris\AppData\Local\Docker\log\vm\kubelet.log and it contained this error message on the final line:

"command failed" err="failed to run Kubelet: invalid configuration: cgroup [\"kubepods\"] has some missing paths: /sys/fs/cgroup/cpu/kubepods"

I think what would be helpful is for the Docker Desktop UI to 1) show a failure state instead of hanging on "starting Kubernetes" and 2) link to the log file directly in the UI. Or maybe to the log directory since I know there are a lot of log files.

I have one WSL Ubuntu-22.04 distro which is the default distro. This is what wsl --list looks like on my machine:

PS C:\Users\chris> wsl --list
Windows Subsystem for Linux Distributions:
Ubuntu-22.04 (Default)
docker-desktop
docker-desktop-data

What fixed it for me was @compadrejunior 's solution of simply

  1. Go to Settings > Resources > WSL Integration
  2. Uncheck Enable integration with my default WSL distro option.

As soon as I unchecked the "enable integration", Kubernetes almost immediately went green. I didn't need to restart anything else.

Since this is a workaround, my assumption is that I am no longer able to use docker or kubernetes commands in my Ubuntu WSL environment - too bad. But I can live with it for the time being.

megahirt avatar May 08 '24 10:05 megahirt

Came across this issue recently when Docker accidentally updated while Kubernetes was in init state. This resulted in corrupted installation state as scripts might not have finished and docker restarted prematurely.

If anyone comes across such invalid state, simple way to fix it is to delete ~/.kube folder and from docker-desktop reset cluster and then change integration with wsl and wait for cluster initialization to finish setting up. This way kubernetes runtime can come online without any error. Distro integration can be reverted afterwards.

If this doesn't work, use clean distro as current distro filesystem may have corrupted files and cannot be recovered. To avoid these issues, best way is to gracefully shutdown docker while it's in critical state or allow setup cleanup so that it can be initialized from step 0.

mxmp210 avatar May 12 '24 07:05 mxmp210