kubernetes-on-arm icon indicating copy to clipboard operation
kubernetes-on-arm copied to clipboard

Published 20 hours ago • verified publisher icon luxas

Curl to nginx/test fails works only once for each node

Open n0npolar opened this issue 8 years ago • 6 comments

Hi!

I installed Kubernetes on ARM via Package Deployment on a present Raspbian install (1 master and 2 worker). Besides modifying /boot/cmdline.txt and resolv.conf (as described in #106) I think I did not do anything different than descibed in the readme.

I end up with: pi@k8sn1:~ $ curl 10.0.0.138

WELCOME TO NGINX

pi@k8sn1:~ $ curl 10.0.0.138

WELCOME TO NGINX

pi@k8sn1:~ $ curl 10.0.0.138

WELCOME TO NGINX

pi@k8sn1:~ $ pi@k8sn1:~ $ curl 10.0.0.138 curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out

I am new to Kubernetes, so I am not sure where to look further. To me it looks as it could be a load balancer problem.

Could you point me in the right direction? Let me know if you need anything else for sorting this out.

Thanks! :-)

n0npolar avatar Jul 03 '16 18:07 n0npolar

Hi, I'm really busy right now, working on a new release Does it still work unreliably? Could you provide more details? kube-config info, proxy logs and the like?

luxas avatar Jul 09 '16 06:07 luxas

Hi!

Yes it still doesn't work reliably:

pi@k8sm:~ $ kubectl get nodes
NAME           STATUS    AGE
192.168.1.81   Ready     24d
192.168.1.82   Ready     24d
192.168.1.83   Ready     24d
pi@k8sm:~ $ kubectl get svc my-nginx
NAME       CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
my-nginx   10.0.0.138   <none>        80/TCP    21d
pi@k8sm:~ $ curl 10.0.0.138
<p>WELCOME TO NGINX</p>
pi@k8sm:~ $ curl 10.0.0.138
curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out
pi@k8sm:~ $ curl 10.0.0.138
curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out
pi@k8sm:~ $ curl 10.0.0.138
<p>WELCOME TO NGINX</p>
pi@k8sm:~ $ curl 10.0.0.138
curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out
pi@k8sm:~ $ curl 10.0.0.138
curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out
pi@k8sm:~ $ curl 10.0.0.138
curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out
pi@k8sm:~ $ curl 10.0.0.138
<p>WELCOME TO NGINX</p>
pi@k8sm:~ $ curl 10.0.0.138
curl: (7) Failed to connect to 10.0.0.138 port 80: Connection timed out

What's also odd, is, that after I booted up the cluster this time (I switched it off after my last post), it seems like not everything was restarted on the first node:

pi@k8sn1:~ $ sudo docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS               NAMES
b0c1a74da4b9        kubernetesonarm/hyperkube   "/hyperkube kubelet -"   14 minutes ago      Up 14 minutes                           k8s-worker
ada9f2e39cd1        kubernetesonarm/hyperkube   "/hyperkube proxy --m"   14 minutes ago      Up 14 minutes                           k8s-worker-proxy

kube-config info (I needed to sudo that) shows:

pi@k8sm:~ $ sudo kube-config info
Architecture: armv7l
Kernel: Linux 4.4.11
CPU: 4 cores x 1200 MHz

Used RAM Memory: 471 MiB
RAM Memory: 925 MiB

Used disk space: 2.1GB (2163676 KB)
Free disk space: 26GB (27148868 KB)

SD Card/deb package was built: 21-03-2016 21:06

kubernetes-on-arm: 
Latest commit: 1d0bbe1
Version: 0.7.0

systemd version: v215
docker version: v1.10.0
kubernetes server version: v1.2.0

CPU Time (minutes):
kubelet: 0:29
kubelet has been up for: 18 minutes
apiserver: 3:26
controller-manager: 0:16
scheduler: 0:01
proxy: 0:01

Proxy logs from node 2:

pi@k8sn2:~ $ sudo docker logs b400534128bd
Flag --resource-container has been deprecated, This feature will be removed in a later release.
I0711 21:10:26.344550       1 iptables.go:177] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: no such file or directory
I0711 21:10:26.409528       1 server.go:178] Running in resource-only container "\"\""
I0711 21:10:26.470653       1 server.go:178] Using iptables Proxier.
I0711 21:10:26.471062       1 server.go:178] Tearing down userspace rules.
I0711 21:10:26.532626       1 conntrack.go:36] Setting nf_conntrack_max to 262144
I0711 21:10:26.532815       1 conntrack.go:41] Setting conntrack hashsize to 65536
I0711 21:10:26.533464       1 conntrack.go:46] Setting nf_conntrack_tcp_timeout_established to 86400
E0711 21:10:26.546184       1 event.go:202] Unable to write event: 'Post http://192.168.1.81:8080/api/v1/namespaces/default/events: dial tcp 192.168.1.81:8080: connection refused' (may retry after sleeping)
E0711 21:10:27.205013       1 event.go:202] Unable to write event: 'Post http://192.168.1.81:8080/api/v1/namespaces/default/events: dial tcp 192.168.1.81:8080: connection refused' (may retry after sleeping)
E0711 21:10:37.206700       1 event.go:202] Unable to write event: 'Post http://192.168.1.81:8080/api/v1/namespaces/default/events: dial tcp 192.168.1.81:8080: connection refused' (may retry after sleeping)
E0711 21:10:47.208323       1 event.go:202] Unable to write event: 'Post http://192.168.1.81:8080/api/v1/namespaces/default/events: dial tcp 192.168.1.81:8080: connection refused' (may retry after sleeping)
I0711 21:10:56.534049       1 proxier.go:593] Not syncing iptables until Services and Endpoints have been received from master
E0711 21:10:57.209780       1 event.go:202] Unable to write event: 'Post http://192.168.1.81:8080/api/v1/namespaces/default/events: dial tcp 192.168.1.81:8080: connection refused' (may retry after sleeping)
E0711 21:11:07.211388       1 event.go:202] Unable to write event: 'Post http://192.168.1.81:8080/api/v1/namespaces/default/events: dial tcp 192.168.1.81:8080: connection refused' (may retry after sleeping)
I0711 21:11:08.661959       1 proxier.go:421] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP
I0711 21:11:08.662643       1 proxier.go:421] Adding new service "default/my-nginx:" at 10.0.0.138:80/TCP
I0711 21:11:08.662881       1 proxier.go:421] Adding new service "kube-system/kube-dns:dns" at 10.0.0.10:53/UDP
I0711 21:11:08.663123       1 proxier.go:421] Adding new service "kube-system/kube-dns:dns-tcp" at 10.0.0.10:53/TCP
I0711 21:11:08.663355       1 proxier.go:421] Adding new service "kube-system/kubernetes-dashboard:" at 10.0.0.168:80/TCP
I0711 21:11:08.663830       1 proxier.go:593] Not syncing iptables until Services and Endpoints have been received from master
I0711 21:11:08.677987       1 proxier.go:462] Setting endpoints for "kube-system/kubernetes-dashboard:" to [10.1.66.3:9090]
I0711 21:11:08.678523       1 proxier.go:462] Setting endpoints for "default/kubernetes:https" to [192.168.1.81:6443]
I0711 21:11:08.678631       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.66.2:80]
I0711 21:11:09.792869       1 proxier.go:462] Removing endpoints for "default/my-nginx:"
I0711 21:11:10.017865       1 proxier.go:462] Removing endpoints for "kube-system/kubernetes-dashboard:"
I0711 21:11:12.727849       1 proxier.go:462] Setting endpoints for "kube-system/kubernetes-dashboard:" to [10.1.66.4:9090]
I0711 21:11:13.347439       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80]
I0711 21:11:13.878802       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.1.66.3:53]
I0711 21:11:13.878938       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns" to [10.1.66.3:53]
I0711 21:11:13.943308       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80 10.1.66.2:80]
I0711 21:17:38.473689       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.66.2:80]
I0711 21:17:48.475895       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80 10.1.66.2:80]
I0711 21:24:20.004840       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80]
I0711 21:24:20.103088       1 proxier.go:462] Removing endpoints for "kube-system/kube-dns:dns-tcp"
I0711 21:24:20.103323       1 proxier.go:462] Removing endpoints for "kube-system/kube-dns:dns"
I0711 21:24:20.162077       1 proxier.go:462] Removing endpoints for "kube-system/kubernetes-dashboard:"
I0711 21:24:21.577398       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.1.66.3:53]
I0711 21:24:21.577621       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns" to [10.1.66.3:53]
I0711 21:24:21.674051       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80 10.1.66.2:80]
I0711 21:24:21.674264       1 proxier.go:462] Setting endpoints for "kube-system/kubernetes-dashboard:" to [10.1.66.4:9090]

Proxy logs from master:

pi@k8sm:~ $ sudo docker logs cd72bfc1a3c8
Flag --resource-container has been deprecated, This feature will be removed in a later release.
I0711 21:10:54.950005       1 iptables.go:177] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: no such file or directory
I0711 21:10:55.050119       1 server.go:178] Running in resource-only container "\"\""
I0711 21:10:55.064808       1 server.go:178] Using iptables Proxier.
I0711 21:10:55.065114       1 server.go:178] Tearing down userspace rules.
I0711 21:10:55.111181       1 conntrack.go:36] Setting nf_conntrack_max to 262144
I0711 21:10:55.111404       1 conntrack.go:41] Setting conntrack hashsize to 65536
I0711 21:10:55.112115       1 conntrack.go:46] Setting nf_conntrack_tcp_timeout_established to 86400
E0711 21:10:55.114784       1 event.go:202] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
E0711 21:10:57.720628       1 event.go:202] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
E0711 21:11:07.723864       1 event.go:202] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
I0711 21:11:09.167098       1 proxier.go:421] Adding new service "kube-system/kube-dns:dns" at 10.0.0.10:53/UDP
I0711 21:11:09.167617       1 proxier.go:421] Adding new service "kube-system/kube-dns:dns-tcp" at 10.0.0.10:53/TCP
I0711 21:11:09.167853       1 proxier.go:421] Adding new service "kube-system/kubernetes-dashboard:" at 10.0.0.168:80/TCP
I0711 21:11:09.168111       1 proxier.go:421] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP
I0711 21:11:09.168319       1 proxier.go:421] Adding new service "default/my-nginx:" at 10.0.0.138:80/TCP
I0711 21:11:09.168622       1 proxier.go:593] Not syncing iptables until Services and Endpoints have been received from master
I0711 21:11:09.175090       1 proxier.go:462] Setting endpoints for "kube-system/kubernetes-dashboard:" to [10.1.66.3:9090]
I0711 21:11:09.175545       1 proxier.go:462] Setting endpoints for "default/kubernetes:https" to [192.168.1.81:6443]
I0711 21:11:09.175636       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.66.2:80]
I0711 21:11:09.796790       1 proxier.go:462] Removing endpoints for "default/my-nginx:"
I0711 21:11:10.016953       1 proxier.go:462] Removing endpoints for "kube-system/kubernetes-dashboard:"
I0711 21:11:12.734789       1 proxier.go:462] Setting endpoints for "kube-system/kubernetes-dashboard:" to [10.1.66.4:9090]
I0711 21:11:13.344736       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80]
I0711 21:11:13.875552       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.1.66.3:53]
I0711 21:11:13.875715       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns" to [10.1.66.3:53]
I0711 21:11:13.951884       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80 10.1.66.2:80]
I0711 21:17:38.477336       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.66.2:80]
I0711 21:17:48.476722       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80 10.1.66.2:80]
I0711 21:24:20.006516       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80]
I0711 21:24:20.131629       1 proxier.go:462] Removing endpoints for "kube-system/kube-dns:dns-tcp"
I0711 21:24:20.131749       1 proxier.go:462] Removing endpoints for "kube-system/kube-dns:dns"
I0711 21:24:20.131805       1 proxier.go:462] Removing endpoints for "kube-system/kubernetes-dashboard:"
I0711 21:24:21.575883       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.1.66.3:53]
I0711 21:24:21.576044       1 proxier.go:462] Setting endpoints for "kube-system/kube-dns:dns" to [10.1.66.3:53]
I0711 21:24:21.667930       1 proxier.go:462] Setting endpoints for "default/my-nginx:" to [10.1.1.2:80 10.1.66.2:80]
I0711 21:24:21.668319       1 proxier.go:462] Setting endpoints for "kube-system/kubernetes-dashboard:" to [10.1.66.4:9090]

Hope this helps...

n0npolar avatar Jul 11 '16 21:07 n0npolar

I ran into the same issue.. at first networking services were working fine but now it seems to be broken:

root@nodejs-2029046627-tjb0b:/srv# curl -v my-nginx
* Rebuilt URL to: my-nginx/
* Hostname was NOT found in DNS cache
*   Trying 10.97.123.170...
^C
root@raspberrypi:/# kubectl get svc --namespace=default
          default
NAME         CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes   10.96.0.1        <none>        443/TCP    1d
mitchpy      10.108.182.146   <none>        8000/TCP   1d
my-nginx     10.97.123.170    <none>        80/TCP     16m
nodejs       10.104.116.166   <none>        3000/TCP   1d
whoami       10.110.232.214   <none>        8000/TCP   1d

All kube-system stuff is running. I set up kubeadm to use flannel. Here is my /etc/resolv.conf:

root@raspberrypi:/# cat /etc/resolv.conf 
# Generated by resolvconf
domain lan
nameserver 192.168.86.1

Any idea what could be the problem?

mitchhh22 avatar Dec 16 '16 20:12 mitchhh22

I actually realized the problem is only with the worker nodes i added to the cluster

mitchhh22 avatar Dec 17 '16 03:12 mitchhh22

So what isn't working? If you're using kubeadm (which you should), please open the issue in that repo

luxas avatar Dec 17 '16 10:12 luxas

@luxas I used kubeadm. It seems to be a problem with restarting a worker node after its joined the cluster.

I opened an issue with some details on kubeadm: #92

mitchhh22 avatar Dec 18 '16 01:12 mitchhh22