minikube icon indicating copy to clipboard operation
minikube copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes

apiServer not starting

Open ps68060 opened this issue 2 years ago • 6 comments

What Happened?

I am new to minikube. I installed it last week and was able to "minikube start" but I was having issue deploying containers.

After lots of stop/start/delete cycles minikube wasn't able to start.

I have since tried "minikube delete", "delete --purge --all" and even uninstalled minikube and deleted anything I could find relating to it and re-installed it a few times.

minikube start fails every time. minikube status shows:

minikube status minikube type: Control Plane host: Running kubelet: Running apiserver: Stopped kubeconfig: Configured

The apiServer will not start.

Attach the log file

logs.txt

Operating System

Windows

Driver

Hyper-V

ps68060 avatar Aug 26 '22 09:08 ps68060

Hi @ps68060, are you still experiencing this issue? It seems that this may be in part why the issue is occurring, and persisting through re-installation:

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Have you tried restarting your computer after uninstalling?

/triage needs-information /kind support

klaases avatar Sep 09 '22 23:09 klaases

I have tried several uninstall processes but I will try:

  1. Uninstall
  2. Reboot
  3. Reinstall

ps68060 avatar Sep 10 '22 09:09 ps68060

This has made things worse:

C:\WINDOWS\system32> minikube start

  • minikube v1.26.0 on Microsoft Windows 10 Pro 10.0.19044 Build 19044
  • Using the hyperv driver based on existing profile
  • Starting control plane node minikube in cluster minikube
  • Creating hyperv VM (CPUs=2, Memory=4000MB, Disk=20000MB) ... ! StartHost failed, but will try again: creating host: create: precreate: no External vswitch nor Default Switch found. A valid vswitch must be available for this command to run. Check https://docs.docker.com/machine/drivers/hyper-v/
  • Creating hyperv VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
  • Failed to start hyperv VM. Running "minikube delete" may fix it: creating host: create: precreate: no External vswitch nor Default Switch found. A valid vswitch must be available for this command to run. Check https://docs.docker.com/machine/drivers/hyper-v/

X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: precreate: no External vswitch nor Default Switch found. A valid vswitch must be available for this command to run. Check https://docs.docker.com/machine/drivers/hyper-v/

ps68060 avatar Sep 12 '22 15:09 ps68060

I uninstalled, rebooted and re-installed again twice and I am now back to the original issue where the first start of minikube gives:

minikube type: Control Plane host: Running kubelet: Running apiserver: Stopped kubeconfig: Misconfigured

minikube update-context

  • "minikube" context has been updated to point to www.xxx.yyy.zzz:8443
  • Current context is "minikube"

and

C:\WINDOWS\system32> minikube start

  • minikube v1.26.0 on Microsoft Windows 10 Pro 10.0.19044 Build 19044
  • Using the hyperv driver based on existing profile
  • Starting control plane node minikube in cluster minikube
  • Updating the running hyperv "minikube" VM ... ! This VM is having trouble accessing https://k8s.gcr.io
  • Preparing Kubernetes v1.24.1 on Docker 20.10.16 ...* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

! Unable to restart cluster, will reset it: apiserver healthz: apiserver process never appeared

  • Generating certificates and keys ...
  • Booting up control plane ... ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem": Process exited with status 1 stdout: [init] Using Kubernetes version: v1.24.1 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/var/lib/minikube/certs" [certs] Using existing ca certificate authority [certs] Using existing apiserver certificate and key on disk [certs] Using existing apiserver-kubelet-client certificate and key on disk [certs] Using existing front-proxy-ca certificate authority [certs] Using existing front-proxy-client certificate and key on disk [certs] Using existing etcd/ca certificate authority [certs] Using existing etcd/server certificate and key on disk [certs] Using existing etcd/peer certificate and key on disk [certs] Using existing etcd/healthcheck-client certificate and key on disk [certs] Using existing apiserver-etcd-client certificate and key on disk [certs] Using the existing "sa" key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred: timed out waiting for the condition

This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl: - 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock logs CONTAINERID'

stderr: W0913 13:42:46.954757 9186 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration! [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster To see the stack trace of this error execute with --v=5 or higher

ps68060 avatar Sep 13 '22 13:09 ps68060

I have updated to 1.27.0 and it is behaving the same.

stderr: W0921 10:52:50.926003 11907 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration! W0921 10:52:50.931105 11907 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /etc/kubelet-resolv.conf [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster To see the stack trace of this error execute with --v=5 or higher

  • Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
  • Related issue: https://github.com/kubernetes/minikube/issues/4172

ps68060 avatar Sep 21 '22 10:09 ps68060

Is it related to this issue ? https://github.com/kubernetes/minikube/issues/5451 #5451

log20220923.txt

ps68060 avatar Sep 23 '22 09:09 ps68060

I believe I have managed to work around the issue by using an older version of kubernetes.

https://github.com/kubernetes/minikube/issues/14477

minikube start --kubernetes-version=v1.23.8

minikube status minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured

ps68060 avatar Sep 23 '22 13:09 ps68060

Hi @ps68060, this is related to the log This VM is having trouble accessing https://k8s.gcr.io/.

If you look at your docker logs you can see that it's continually failing to pull images as it has not internet connectivity, which is causing the start to fail:

Aug 26 08:47:47 minikube dockerd[946]: time="2022-08-26T08:47:47.139378400Z" level=warning msg="Error getting v2 registry: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:57760->172.20.224.1:53: i/o timeout"
Aug 26 08:47:47 minikube dockerd[946]: time="2022-08-26T08:47:47.139405700Z" level=error msg="Not continuing with pull after error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:57760->172.20.224.1:53: i/o timeout"
Aug 26 08:47:47 minikube dockerd[946]: time="2022-08-26T08:47:47.139455400Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:57760->172.20.224.1:53: i/o timeout"
Aug 26 08:48:47 minikube dockerd[946]: time="2022-08-26T08:48:47.146308500Z" level=warning msg="Error getting v2 registry: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:43580->172.20.224.1:53: i/o timeout"
Aug 26 08:48:47 minikube dockerd[946]: time="2022-08-26T08:48:47.146336800Z" level=info msg="Attempting next endpoint for pull after error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:43580->172.20.224.1:53: i/o timeout"
Aug 26 08:48:47 minikube dockerd[946]: time="2022-08-26T08:48:47.155022900Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:43580->172.20.224.1:53: i/o timeout"
Aug 26 08:49:07 minikube dockerd[946]: time="2022-08-26T08:49:07.148901500Z" level=warning msg="Error getting v2 registry: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:59581->172.20.224.1:53: i/o timeout"
Aug 26 08:49:07 minikube dockerd[946]: time="2022-08-26T08:49:07.148927100Z" level=error msg="Not continuing with pull after error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:59581->172.20.224.1:53: i/o timeout"
Aug 26 08:49:07 minikube dockerd[946]: time="2022-08-26T08:49:07.148976700Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:59581->172.20.224.1:53: i/o timeout"
Aug 26 08:49:27 minikube dockerd[946]: time="2022-08-26T08:49:27.150867000Z" level=warning msg="Error getting v2 registry: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:49330->172.20.224.1:53: i/o timeout"
Aug 26 08:49:27 minikube dockerd[946]: time="2022-08-26T08:49:27.150894300Z" level=error msg="Not continuing with pull after error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:49330->172.20.224.1:53: i/o timeout"
Aug 26 08:49:27 minikube dockerd[946]: time="2022-08-26T08:49:27.150944300Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get \"https://k8s.gcr.io/v2/\": dial tcp: lookup k8s.gcr.io on 172.20.224.1:53: read udp 172.20.226.222:49330->172.20.224.1:53: i/o timeout"

If you use the --container-runtime=containerd that should help with minikube starting, but you still won't be able to pull in any remote images. Are you using a proxy or VPN?

spowelljr avatar Oct 12 '22 18:10 spowelljr

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 10 '23 19:01 k8s-triage-robot

Hi @ps68060, we haven't heard back from you, do you still have this issue? There isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.

I will close this issue for now but feel free to reopen when you feel ready to provide more information.

spowelljr avatar Jan 10 '23 19:01 spowelljr