kismatic
kismatic copied to clipboard
Published
20 hours ago •
apprenda
apprenda
Start Kubernetes DNS error
Installing Cluster==================================================================
Configure Cluster Prerequisites [OK]
Gather Node Facts [OK]
Copy Additional Files and Directories [SKIPPED]
Update Hosts File [SKIPPED]
Deploy Cluster Certificates [OK]
Generate Kubectl Config File [OK]
Deploy Etcd Certificates [OK]
Configure Package Repositories [OK]
Install Docker [OK]
Start Kubernetes Etcd Cluster [OK]
Start Network Etcd Cluster [OK]
Start Kubernetes Kubelet [OK]
Start Kubernetes API Server [OK]
Start Kubernetes Scheduler [OK]
Start Kubernetes Controller Manager [OK]
Validate Kubernetes Control Plane is Running [OK]
Start Kubernetes Proxy [OK]
Label and Taint Kubernetes Nodes [OK]
Start Calico Network Components [OK]
Validate Calico Network Components [OK]
Configure Calico Network Policy [OK]
Start Weave Network Components [SKIPPED]
Validate Weave Network Components [SKIPPED]
Start Contiv Network Components [SKIPPED]
Start Kubernetes Pod Rescheduler [OK]
Start Kubernetes DNS [ERROR]
- Task: fail if DNS pod validation command could not determine if the DNS pods are ready
was01: Waited for all DNS pods to be ready, but they took longer than 5 minutes to be in the ready state.
[ERROR]
error installing: error running playbook: error running ansible: exit status 2
I logged in to was01 I am running
kismaticuser@was01:~/kubernate$
kismaticuser@was01:~/kubernate$ /usr/bin/kubectl get pods
error: the server doesn't have a resource type "po
```ds"
the server doesn't have a resource type "pods"
Something seems really wrong, I would try running kismatic reset and try the the install again.
Again DNS issue
broker01@broker01:~$ sudo kubectl --kubeconfig /root/.kube/config get pods --namespace kube-
system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-d7fb79cc4-9rwpw 1/1 Running 0 11m
calico-node-bgj8v 2/2 Running 0 11m
calico-node-s6ggp 2/2 Running 0 11m
calico-node-sj7xz 2/2 Running 0 11m
kube-apiserver-was01 1/1 Running 1 43m
kube-controller-manager-was01 1/1 Running 0 42m
kube-dns-5b9dbbdcc6-dhvl6 2/3 CrashLoopBackOff 10 9m
kube-dns-5b9dbbdcc6-s4xn5 2/3 CrashLoopBackOff 10 9m
kube-proxy-f727m 1/1 Running 0 21m
kube-proxy-jztr6 1/1 Running 0 21m
kube-proxy-knpgz 1/1 Running 0 21m
kube-scheduler-was01 1/1 Running 0 44m
rescheduler-was01 1/1 Running 0 10m
DNS is getting crashloopbackoff
on describing dns pod it is giving following error
broker01@broker01:~$ sudo kubectl --kubeconfig /root/.kube/config describe pod kube-dns-5b9dbbdcc6-dhvl6 --namespace kube-system
Name: kube-dns-5b9dbbdcc6-dhvl6
Namespace: kube-system
Node: db01/192.168.0.70
Start Time: Wed, 09 May 2018 23:55:27 +0530
Labels: k8s-app=kube-dns
pod-template-hash=1658668772
Annotations: scheduler.alpha.kubernetes.io/critical-pod=
Status: Running
IP: 172.16.218.194
Controlled By: ReplicaSet/kube-dns-5b9dbbdcc6
Containers:
kubedns:
Container ID: docker://200a4785a0b5237b1b80c9eec4dc36750e45c9a386df1a41d35ce12f6272e155
Image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10
Image ID: docker-pullable://gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:b99fc3eee2a9f052f7eb4cc00f15eb12fc405fa41019baa2d6b79847ae7284a8
Ports: 10053/UDP, 10053/TCP, 10055/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
--domain=cluster.local
--dns-port=10053
--config-dir=/kube-dns-config
--v=2
State: Running
Started: Thu, 10 May 2018 00:05:27 +0530
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Thu, 10 May 2018 00:02:29 +0530
Finished: Thu, 10 May 2018 00:03:29 +0530
Ready: False
Restart Count: 5
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
Environment:
KUBERNETES_SERVICE_HOST: was01
KUBERNETES_SERVICE_PORT: 6443
PROMETHEUS_PORT: 10055
Mounts:
/kube-dns-config from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-kjzzw (ro)
dnsmasq:
Container ID: docker://9249a94a3bc6f907efb4bbbc681f733ef13b32d288d2ae74e47392ac43a6461b
Image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10
Image ID: docker-pullable://gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:bbb2a290a568125b3b996028958eb773f33b5b87a6b37bf38a28f8b62dddb3c8
Ports: 53/UDP, 53/TCP
Host Ports: 0/UDP, 0/TCP
Args:
-v=2
-logtostderr
-configDir=/etc/k8s/dns/dnsmasq-nanny
-restartDnsmasq=true
--
-k
--cache-size=1000
--log-facility=-
--server=/cluster.local/127.0.0.1#10053
--server=/in-addr.arpa/127.0.0.1#10053
--server=/ip6.arpa/127.0.0.1#10053
State: Running
Started: Thu, 10 May 2018 00:05:28 +0530
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Thu, 10 May 2018 00:03:27 +0530
Finished: Thu, 10 May 2018 00:05:27 +0530
Ready: True
Restart Count: 4
Requests:
cpu: 150m
memory: 20Mi
Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-kjzzw (ro)
sidecar:
Container ID: docker://f4c7228f41bb9c05c8c02e71f34c1f6016e0c38af91708a9dd6e62938f2b0681
Image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10
Image ID: docker-pullable://gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:4f1ab957f87b94a5ec1edc26fae50da2175461f00afecf68940c4aa079bd08a4
Port: 10054/TCP
Host Port: 0/TCP
Args:
--v=2
--logtostderr
--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 10 May 2018 00:03:54 +0530
Finished: Thu, 10 May 2018 00:05:29 +0530
Ready: False
Restart Count: 3
Requests:
cpu: 10m
memory: 20Mi
Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-kjzzw (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
kube-dns-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-dns
Optional: true
kube-dns-token-kjzzw:
Type: Secret (a volume populated by a Secret)
SecretName: kube-dns-token-kjzzw
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned kube-dns-5b9dbbdcc6-dhvl6 to db01
Normal SuccessfulMountVolume 10m kubelet, db01 MountVolume.SetUp succeeded for volume "kube-dns-config"
Normal SuccessfulMountVolume 10m kubelet, db01 MountVolume.SetUp succeeded for volume "kube-dns-token-kjzzw"
Normal Pulling 10m kubelet, db01 pulling image "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10"
Normal Pulled 9m kubelet, db01 Successfully pulled image "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10"
Normal Pulling 9m kubelet, db01 pulling image "gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10"
Normal Pulled 8m kubelet, db01 Successfully pulled image "gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10"
Normal Pulling 8m kubelet, db01 pulling image "gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10"
Normal Started 8m kubelet, db01 Started container
Normal Created 8m kubelet, db01 Created container
Normal Pulled 8m kubelet, db01 Successfully pulled image "gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10"
Normal Created 8m kubelet, db01 Created container
Normal Started 8m kubelet, db01 Started container
Normal Started 8m (x2 over 9m) kubelet, db01 Started container
Normal Created 8m (x2 over 9m) kubelet, db01 Created container
Normal Pulled 8m kubelet, db01 Container image "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10" already present on machine
Warning Unhealthy 7m (x5 over 8m) kubelet, db01 Readiness probe failed: Get http://172.16.218.194:8081/readiness: dial tcp 172.16.218.194:8081: getsockopt: connection refused
Warning Unhealthy 5m (x7 over 7m) kubelet, db01 Liveness probe failed: HTTP probe failed with statuscode: 503
Warning Unhealthy 28s kubelet, db01 Liveness probe failed: Get http://172.16.218.194:10054/metrics: dial tcp 172.16.218.194:10054: getsockopt: connection refused
Seems issue with kubedns pod image, I have googled that image 1.14.10 probe pointing to port 443 instead of 6443, can you consider this as bug and fix the kismatic in next release?
Is there anything in the logs?
kubectl logs kube-dns-5b9dbbdcc6-dhvl6 -n kube-system -c kubedns --previous
kismaticuser@was01:~/kubernate$ sudo kubectl --kubeconfig /root/.kube/config logs kube- dns-6d4bcb6669-s45f7 -n kube-system -c kubedns --previous
I0511 09:41:53.382207 1 dns.go:48] version: 1.14.8
I0511 09:41:53.384741 1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0511 09:41:53.384832 1 server.go:119] FLAG: --alsologtostderr="false"
I0511 09:41:53.384865 1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0511 09:41:53.384883 1 server.go:119] FLAG: --config-map=""
I0511 09:41:53.384896 1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0511 09:41:53.384908 1 server.go:119] FLAG: --config-period="10s"
I0511 09:41:53.384924 1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0511 09:41:53.384937 1 server.go:119] FLAG: --dns-port="10053"
I0511 09:41:53.384954 1 server.go:119] FLAG: --domain="cluster.local."
I0511 09:41:53.384975 1 server.go:119] FLAG: --federations=""
I0511 09:41:53.384992 1 server.go:119] FLAG: --healthz-port="8081"
I0511 09:41:53.385005 1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0511 09:41:53.385021 1 server.go:119] FLAG: --kube-master-url=""
I0511 09:41:53.385036 1 server.go:119] FLAG: --kubecfg-file=""
I0511 09:41:53.385049 1 server.go:119] FLAG: --log-backtrace-at=":0"
I0511 09:41:53.385071 1 server.go:119] FLAG: --log-dir=""
I0511 09:41:53.385085 1 server.go:119] FLAG: --log-flush-frequency="5s"
I0511 09:41:53.385098 1 server.go:119] FLAG: --logtostderr="true"
I0511 09:41:53.385113 1 server.go:119] FLAG: --nameservers=""
I0511 09:41:53.385125 1 server.go:119] FLAG: --stderrthreshold="2"
I0511 09:41:53.385141 1 server.go:119] FLAG: --v="2"
I0511 09:41:53.385156 1 server.go:119] FLAG: --version="false"
I0511 09:41:53.385176 1 server.go:119] FLAG: --vmodule=""
I0511 09:41:53.385312 1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0511 09:41:53.385761 1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0511 09:41:53.385799 1 dns.go:146] Starting endpointsController
I0511 09:41:53.385813 1 dns.go:149] Starting serviceController
I0511 09:41:53.387625 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0511 09:41:53.387678 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0511 09:41:53.886225 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:54.386295 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:54.886220 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:55.386327 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:55.886180 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:56.386203 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:56.886209 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:57.386180 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:57.886233 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:58.386222 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:58.886370 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:59.386206 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:41:59.886202 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:00.386294 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:00.886308 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:01.386370 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:01.886295 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:02.386251 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:02.886310 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:03.386173 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0511 09:42:03.388471 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://was01:6443/api/v1/services?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:59136->127.0.1.1:53: i/o timeout
E0511 09:42:03.388471 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://was01:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:59136->127.0.1.1:53: i/o timeout
I0511 09:42:03.886312 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:04.386236 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:04.886442 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:05.386127 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:05.886203 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:06.386215 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:06.886300 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:07.386173 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:07.886139 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:08.386115 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:08.886169 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:09.386368 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:09.886120 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:10.386227 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:10.886190 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:11.386237 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:11.886214 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:12.386302 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:12.886178 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:13.386211 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:13.886187 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:14.386153 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0511 09:42:14.390079 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://was01:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:59483->127.0.1.1:53: i/o timeout
E0511 09:42:14.390147 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://was01:6443/api/v1/services?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:59483->127.0.1.1:53: i/o timeout
I0511 09:42:14.886223 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:15.386374 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:15.886149 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:16.386333 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:16.886334 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:17.386239 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:17.886189 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:18.386195 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:18.886310 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:19.386323 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:19.886361 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:20.386313 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:20.886224 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:21.386210 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:21.886319 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:22.386192 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:22.886234 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:23.386165 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:23.886176 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:24.386248 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:24.886336 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:25.386301 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0511 09:42:25.391682 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://was01:6443/api/v1/services?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:43142->127.0.1.1:53: i/o timeout
E0511 09:42:25.391748 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://was01:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:43142->127.0.1.1:53: i/o timeout
I0511 09:42:25.886185 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:26.386259 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:26.886238 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:27.386186 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:27.886159 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:28.386148 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:28.886155 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:29.386237 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:29.886223 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:30.386318 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:30.886309 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:31.386204 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:31.886193 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:32.386174 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:32.886203 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:33.386241 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:33.886494 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:34.386232 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:34.886230 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:35.386197 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:35.886195 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:36.386318 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0511 09:42:36.393097 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://was01:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:40976->127.0.1.1:53: i/o timeout
E0511 09:42:36.393126 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://was01:6443/api/v1/services?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:40976->127.0.1.1:53: i/o timeout
I0511 09:42:36.886193 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:37.386343 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:37.886134 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:38.386165 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:38.886231 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:39.386333 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:39.886197 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:40.386243 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:40.886126 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:41.386326 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:41.886337 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:42.386331 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:42.886183 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:43.386202 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:43.886327 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:44.386326 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:44.886381 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:45.386225 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:45.886399 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:46.386237 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:46.886325 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:47.386237 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0511 09:42:47.394715 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://was01:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:33000->127.0.1.1:53: i/o timeout
E0511 09:42:47.394715 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://was01:6443/api/v1/services?resourceVersion=0: dial tcp: lookup was01 on 127.0.1.1:53: read udp 127.0.0.1:33000->127.0.1.1:53: i/o timeout
I0511 09:42:47.886256 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:48.386222 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:48.886231 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:49.386206 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:49.886313 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:50.386189 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:50.886328 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:51.386236 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:51.886214 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:52.386196 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0511 09:42:52.886421 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
F0511 09:42:53.386138 1 dns.go:167] Timeout waiting for initialization
How does was01 is resolved, is it part of the actual DNS or is it in /etc/hosts?
I have added entry in /etc/hosts for all two nodes. was01 (master) db1 (worker) Operating system is ubuntu 16.04, it there any port forwarding issue in ubuntu?
In the plan file can you set update_hosts_files: true and retry the installation, it would probably be best to run kismatic reset or new VMs
I am getting same issue with Kismatic 1.11 My hosts are RHEL 7.5
The kubedns pod goes into Crashloopbackoff state , same as described by @shubhanshushukla
I have already tried with : - update_hosts_files: true
How are the kube-proxy and the CNI pods are they restarting, check the logs for both to see if there is anything interesting.
The times I've seen DNS crashing is when there are issues with the network.
Hi @dkoshkin , I am again getting same error, even after setting update host-file to true Please find below logs
was01@was01:~$ sudo kubectl --kubeconfig /root/.kube/config logs kube-dns-5dfd8b4b8f-4dspc -n kube-system -c kubedns --previous
I0701 07:32:19.834858 1 dns.go:48] version: 1.14.8
I0701 07:32:19.837046 1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0701 07:32:19.837117 1 server.go:119] FLAG: --alsologtostderr="false"
I0701 07:32:19.837144 1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0701 07:32:19.837159 1 server.go:119] FLAG: --config-map=""
I0701 07:32:19.837175 1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0701 07:32:19.837187 1 server.go:119] FLAG: --config-period="10s"
I0701 07:32:19.837204 1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0701 07:32:19.837215 1 server.go:119] FLAG: --dns-port="10053"
I0701 07:32:19.837230 1 server.go:119] FLAG: --domain="cluster.local."
I0701 07:32:19.837246 1 server.go:119] FLAG: --federations=""
I0701 07:32:19.837268 1 server.go:119] FLAG: --healthz-port="8081"
I0701 07:32:19.837280 1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0701 07:32:19.837292 1 server.go:119] FLAG: --kube-master-url=""
I0701 07:32:19.837305 1 server.go:119] FLAG: --kubecfg-file=""
I0701 07:32:19.837316 1 server.go:119] FLAG: --log-backtrace-at=":0"
I0701 07:32:19.837337 1 server.go:119] FLAG: --log-dir=""
I0701 07:32:19.837350 1 server.go:119] FLAG: --log-flush-frequency="5s"
I0701 07:32:19.837361 1 server.go:119] FLAG: --logtostderr="true"
I0701 07:32:19.837376 1 server.go:119] FLAG: --nameservers=""
I0701 07:32:19.837387 1 server.go:119] FLAG: --stderrthreshold="2"
I0701 07:32:19.837398 1 server.go:119] FLAG: --v="2"
I0701 07:32:19.837409 1 server.go:119] FLAG: --version="false"
I0701 07:32:19.837432 1 server.go:119] FLAG: --vmodule=""
I0701 07:32:19.837583 1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0701 07:32:19.838072 1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0701 07:32:19.838097 1 dns.go:146] Starting endpointsController
I0701 07:32:19.838108 1 dns.go:149] Starting serviceController
I0701 07:32:19.838263 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0701 07:32:19.838309 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0701 07:32:20.338423 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:20.838428 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:21.338568 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:21.838484 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:22.338479 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:22.838556 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:23.338458 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:23.838473 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:24.338453 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:24.838395 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:25.338484 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:25.838578 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:26.338497 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:26.838522 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:27.338502 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:27.838478 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:28.338475 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:28.838493 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:29.338521 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:29.838426 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0701 07:32:29.843251 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:35032->127.0.1.1:53: i/o timeout
E0701 07:32:29.843260 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://stg.generalmobi.mobi:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:35032->127.0.1.1:53: i/o timeout
I0701 07:32:30.338482 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:30.838464 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:31.338753 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:31.838579 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:32.338474 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:32.838488 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:33.338456 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:33.838449 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:34.338451 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:34.838436 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:35.338419 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:35.838550 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:36.338474 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:36.838495 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:37.338421 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:37.838526 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:38.338506 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:38.838442 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:39.338481 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:39.838406 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:40.338510 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:40.838509 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0701 07:32:40.844611 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://stg.generalmobi.mobi:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:57531->127.0.1.1:53: i/o timeout
E0701 07:32:40.844632 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:57531->127.0.1.1:53: i/o timeout
I0701 07:32:41.338546 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:41.838486 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:42.338495 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:42.838526 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:43.338541 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:43.838419 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:44.338511 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:44.838460 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:45.338498 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:45.838498 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:46.338562 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:46.838475 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:47.338454 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:47.838449 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:48.338441 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:48.838458 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:49.338488 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:49.838384 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:50.338454 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:50.838571 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:51.338426 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:51.838461 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0701 07:32:51.845791 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:54368->127.0.1.1:53: i/o timeout
E0701 07:32:51.845820 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://stg.generalmobi.mobi:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:54368->127.0.1.1:53: i/o timeout
I0701 07:32:52.338489 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:52.838430 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:53.338537 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:53.838540 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:54.338624 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:54.838407 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:55.338492 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:55.838544 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:56.338443 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:56.838497 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:57.338420 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:57.838524 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:58.338436 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:58.838556 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:59.338490 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:32:59.838422 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:00.338489 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:00.838469 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:01.338514 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:01.838505 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:02.338473 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:02.838478 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0701 07:33:02.847153 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:37697->127.0.1.1:53: i/o timeout
E0701 07:33:02.847153 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://stg.generalmobi.mobi:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:37697->127.0.1.1:53: i/o timeout
I0701 07:33:03.338465 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:03.838491 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:04.338418 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:04.838458 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:05.338533 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:05.838470 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:06.338499 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:06.838480 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:07.338481 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:07.838507 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:08.338422 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:08.838504 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:09.338549 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:09.838500 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:10.338445 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:10.838504 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:11.338575 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:11.838417 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:12.338455 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:12.838507 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:13.338452 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:13.838514 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
E0701 07:33:13.848561 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://stg.generalmobi.mobi:6443/api/v1/endpoints?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:39763->127.0.1.1:53: i/o timeout
E0701 07:33:13.848581 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0: dial tcp: lookup stg.generalmobi.mobi on 127.0.1.1:53: read udp 127.0.0.1:39763->127.0.1.1:53: i/o timeout
I0701 07:33:14.338424 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:14.838471 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:15.338458 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:15.838468 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:16.338533 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:16.838461 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:17.338470 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:17.838410 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:18.338586 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:18.838444 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0701 07:33:19.338571 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
F0701 07:33:19.838430 1 dns.go:167] Timeout waiting for initialization
Log read udp 127.0.0.1:39763->127.0.1.1:53: i/o timeout makes me think your existing DNS is not properly setup and the master LB cannot be properly resolved.
Try curling https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0 from a worker node to see if that works.
I would also recommend upgrading to the latest release of KET, the add-ons we're modifed to connect to the apiserver via internal IP.
CURL Response
was01@was01:~$ curl -k "https://stg.generalmobi.mobi:6443/api/v1/services?resourceVersion=0"
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services is forbidden: User \"system:anonymous\" cannot list services at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "services"
},
"code": 403
}
Do you think below command will help?
iptables -P FORWARD ACCEPT
Hi @dkoshkin I have update kismatic to 1.12, Installation completed successfully. Got this output
Smoke Test Master Node Finished
The cluster was installed successfully!
- To use the generated kubeconfig file with kubectl:
* use "./kubectl --kubeconfig generated/kubeconfig"
* or copy the config file "cp generated/kubeconfig ~/.kube/config"
- To view the Kubernetes dashboard: "./kismatic dashboard"
- To SSH into a cluster node: "./kismatic ssh etcd|master|worker|storage|$node.host"
I run
./kismatic dashboard
Opening kubernetes dashboard in default browser...
Use the kubeconfig in "generated/dashboard-admin-kubeconfig"
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: www-browser: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: links2: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: elinks: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: links: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: lynx: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: w3m: not found
xdg-open: no method available for opening 'http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login'
Unexpected error opening the kubernetes dashboard: exit status 3. You may access it at "http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login"Starting to serve on 127.0.0.1:8001
I can access by forwarding 8001 to external IPL:9001, but getting below error
Let me know how I can open dashboard
Hi @dkoshkin ,
I run below command proxy
kubectl proxy --kubeconfig generated/kubeconfig --address 0.0.0.0 --accept-hosts '.*'
Now dashboard is getting timed out get below response in browser:
Error: 'dial tcp 172.16.70.3:8443: getsockopt: connection timed out'
Trying to reach: 'https://172.16.70.3:8443/'
Dashboard logs also throwing below messages
./kubectl --kubeconfig generated/kubeconfig logs kubernetes-dashboard-b6db5b8d9-l865t -n kube-system
2018/07/01 17:28:04 Starting overwatch
2018/07/01 17:28:04 Using in-cluster config to connect to apiserver
2018/07/01 17:28:04 Using service account token for csrf signing
2018/07/01 17:28:04 No request provided. Skipping authorization
2018/07/01 17:28:04 Successful initial request to the apiserver, version: v1.10.2
2018/07/01 17:28:04 Generating JWE encryption key
2018/07/01 17:28:04 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/07/01 17:28:04 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/07/01 17:28:12 Storing encryption key in a secret
2018/07/01 17:28:12 Creating in-cluster Heapster client
2018/07/01 17:28:13 Auto-generating certificates
2018/07/01 17:28:13 Successfully created certificates
2018/07/01 17:28:13 Serving securely on HTTPS port: 8443
2018/07/01 17:30:22 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.218.195:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.218.195:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:33:01 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.218.195:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.218.195:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:35:41 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.218.195:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.218.195:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:38:21 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.218.195:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.218.195:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:41:01 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.218.195:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.218.195:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:43:40 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.70.2:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.70.2:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:46:20 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.70.2:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.70.2:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
2018/07/01 17:49:00 Metric client health check failed: an error on the server ("Error: 'dial tcp 172.16.218.195:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.16.218.195:8082/healthz'") has prevented the request from succeeding (get services heapster). Retrying in 30 seconds.
kismaticuser@was01:~/kismatic112$ ./kubectl --kubeconfig generated/kubeconfig cluster-info
Kubernetes master is running at https://stg.generalmobi.mobi:6443
KubeDNS is running at https://stg.generalmobi.mobi:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Raised new BUG https://github.com/apprenda/kismatic/issues/1263 , DNS issue resolved, so can you close this.