kubedns插件部署失败, node节点iptables提示dns has no endpoints

[ericjee]

新配集群时遇见，master机器kubedns无法create pods，求助

master节点: kubectl get pods --all-namespaces -o wide

NAMESPACE   NAME                        READY     STATUS    RESTARTS   AGE       IP            
default     my-nginx-3418754612-3wdp7   1/1       Running   0          7d        172.30.65.3   
default     my-nginx-3418754612-ddz0v   1/1       Running   0          7d        172.30.65.4   
default     nginx-ds-vsgk0              1/1       Running   0          12d       172.30.65.2

kubectl get services --all-namespaces

NAMESPACE     NAME         CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   10.254.0.1       <none>        443/TCP         13d
default       nginx-ds     10.254.148.239   <nodes>       80:8571/TCP     12d
kube-system   kube-dns     10.254.0.2       <none>        53/UDP,53/TCP   8d

node节点： iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-ISOLATION  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (1 references)
target     prot opt source               destination         
REJECT     udp  --  anywhere             10.254.0.2           /* kube-system/kube-dns:dns has no endpoints */ udp dpt:domain reject-with icmp-port-unreachable
REJECT     tcp  --  anywhere             10.254.0.2           /* kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:domain reject-with icmp-port-unreachable

[opsnull]

出错信息贴出来看看。

[ericjee]

@opsnull kubectl cluster-info

Kubernetes master is running at https://$master:6443
KubeDNS is running at https://$master:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns

浏览器访问 User "system:anonymous" cannot proxy services in the namespace "kube-system".

kubectl get ep kube-dns

Error from server (NotFound): endpoints "kube-dns" not found

[ericjee]

node节点 systemctl status kube-proxy

● kube-proxy.service - Kubernetes Kube-Proxy Server
   Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-06-21 18:44:12 CST; 1 weeks 6 days ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 13239 (kube-proxy)
   Memory: 12.5M
   CGroup: /system.slice/kube-proxy.service
           └─13239 /usr/local/bin/kube-proxy --bind-address=nodeip --hostname-override=nodeip --cluster-cidr=10.254.0.0/16 --kubecon...

Jun 21 18:51:57 ub-opstools01 kube-proxy[13239]: I0621 18:51:57.566512   13239 proxier.go:472] Adding new service "default/nginx-ds:htt...:80/TCP
Jun 21 18:51:57 ub-opstools01 kube-proxy[13239]: I0621 18:51:57.579084   13239 proxier.go:1374] Opened local port "nodePort for default...71/tcp)
Jul 05 10:38:30 ub-opstools01 kube-proxy[13239]: I0705 10:38:30.062878   13239 proxier.go:497] Removing service "kube-system/kube-dns:dns"
Jul 05 10:38:30 ub-opstools01 kube-proxy[13239]: I0705 10:38:30.062916   13239 proxier.go:497] Removing service "kube-system/kube-dns:dns-tcp"
Jul 05 10:38:30 ub-opstools01 kube-proxy[13239]: I0705 10:38:30.077540   13239 conntrack.go:36] Deleting connection tracking state for ...254.0.2
Jul 05 10:38:30 ub-opstools01 kube-proxy[13239]: E0705 10:38:30.077611   13239 conntrack.go:42] conntrack returned error: error looking...n $PATH
Jul 05 10:38:30 ub-opstools01 kube-proxy[13239]: I0705 10:38:30.290332   13239 proxier.go:472] Adding new service "kube-system/kube-dns...:53/UDP
Jul 05 10:38:30 ub-opstools01 kube-proxy[13239]: I0705 10:38:30.290384   13239 proxier.go:472] Adding new service "kube-system/kube-dns...:53/TCP
Hint: Some lines were ellipsized, use -l to show in full.

[opsnull]

conntrack returned error: error looking...n $PATH

系统缺少了 iptables 命令？

[richzzp]

请问这个问题最后怎么解决的呢？ 我一样是没有endpoints， dns的相关pod都没启动

[richzzp]

root@k8s1-master:~/kubernetes/cluster/addons/dns# kubectl describe deployment kube-dns -n kube-system
Name: kube-dns Namespace: kube-system CreationTimestamp: Tue, 26 Dec 2017 09:35:15 +0000 Labels: addonmanager.kubernetes.io/mode=Reconcile k8s-app=kube-dns kubernetes.io/cluster-service=true Annotations: deployment.kubernetes.io/revision=1 Selector: k8s-app=kube-dns Replicas: 1 desired | 0 updated | 0 total | 0 available | 1 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 0 max unavailable, 10% max surge Pod Template: Labels: k8s-app=kube-dns Annotations: scheduler.alpha.kubernetes.io/critical-pod= Service Account: kube-dns Containers: kubedns: Image: xuejipeng/k8s-dns-kube-dns-amd64:v1.14.1 Ports: 10053/UDP, 10053/TCP, 10055/TCP Args: --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2 Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3 Environment: PROMETHEUS_PORT: 10055 Mounts: /kube-dns-config from kube-dns-config (rw) dnsmasq: Image: xuejipeng/k8s-dns-dnsmasq-nanny-amd64:v1.14.1 Ports: 53/UDP, 53/TCP Args: -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --log-facility=- --server=/cluster.local./127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053 Requests: cpu: 150m memory: 20Mi Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: Mounts: /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw) sidecar: Image: xuejipeng/k8s-dns-sidecar-amd64:v1.14.1 Port: 10054/TCP Args: --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local.,5,A --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local.,5,A Requests: cpu: 10m memory: 20Mi Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: Mounts: Volumes: kube-dns-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: kube-dns Optional: true Conditions: Type Status Reason

---

Available False MinimumReplicasUnavailable ReplicaFailure True FailedCreate OldReplicaSets: NewReplicaSet: kube-dns-699984412 (0/1 replicas created) Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

1m 1m 1 deployment-controller Normal ScalingReplicaSet Scaled up replica set kube-dns-699984412 to 1

[ikingye]

@opsnull kubectl cluster-info

Kubernetes master is running at https://$master:6443
KubeDNS is running at https://$master:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns

浏览器访问 User "system:anonymous" cannot proxy services in the namespace "kube-system".

kubectl get ep kube-dns

Error from server (NotFound): endpoints "kube-dns" not found

kubectl get ep kube-dns -n kube-system