microk8s icon indicating copy to clipboard operation
microk8s copied to clipboard
verified publisher icon canonical

10.152.183.1 is not reachable on the slave-node

Open godlzr opened this issue 4 years ago • 4 comments

microk8s inspect

Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
 FAIL:  Service snap.microk8s.daemon-apiserver-kicker is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-apiserver-kicker
  Service snap.microk8s.daemon-kubelite is running
  Service snap.microk8s.daemon-flanneld is running
 FAIL:  Service snap.microk8s.daemon-etcd is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-etcd
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

Building the report tarball
  Report tarball is at /var/snap/microk8s/2552/inspection-report-20211007_200543.tar.gz

inspection-report-20211007_200543.tar.gz

I have two oracle instances with 3 OCPU and 18G RAM, the single-cluster is working fine on each of them, but after I make them a cluster, the slave node is having network issue. For example, I enabled the ingress, and the ingress pod on the slave-node crashes while creating API client on https://10.152.183.1:443

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0-alpha.2
  Build:         cbd63861922603f85853f0ce66f959cde4d81382
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

I1007 20:06:31.747741       8 flags.go:211] "Watching for Ingress" class="public"
W1007 20:06:31.747825       8 flags.go:214] Only Ingresses with class "public" will be processed by this Ingress controller
W1007 20:06:31.748128       8 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1007 20:06:31.748312       8 main.go:234] "Creating API client" host="https://10.152.183.1:443"

I the network is not reachable in busybox as welll

/ # telnet 10.152.183.1 443
telnet: can't connect to remote host (10.152.183.1): No route to host

the iptables looks like

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N InstanceServices
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-KUBELET-CANARY
-N KUBE-NODEPORTS
-N KUBE-PROXY-CANARY
-N KUBE-SERVICES
-N cali-FORWARD
-N cali-INPUT
-N cali-OUTPUT
-N cali-cidr-block
-N cali-from-hep-forward
-N cali-from-host-endpoint
-N cali-from-wl-dispatch
-N cali-to-hep-forward
-N cali-to-host-endpoint
-N cali-to-wl-dispatch
-N cali-wl-to-host
-A INPUT -m comment --comment "cali:Cz_u1IQiXIMmKD4c" -j cali-INPUT
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -j KUBE-FIREWALL
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -m comment --comment "cali:wUHhoiAYhphO9Mso" -j cali-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -s 10.1.0.0/16 -j ACCEPT
-A FORWARD -d 10.1.0.0/16 -j ACCEPT
-A FORWARD -s 10.1.0.0/16 -m comment --comment "generated for MicroK8s pods" -j ACCEPT
-A FORWARD -d 10.1.0.0/16 -m comment --comment "generated for MicroK8s pods" -j ACCEPT
-A FORWARD -m comment --comment "cali:S93hcgKJrXEqnTfs" -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A FORWARD -m comment --comment "cali:mp77cMpurHhyjLrM" -j MARK --set-xmark 0x10000/0x10000
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-FORWARD -m comment --comment "cali:vjrMJCRpqwy5oRoX" -j MARK --set-xmark 0x0/0xe0000
-A cali-FORWARD -m comment --comment "cali:A_sPAO0mcxbT9mOV" -m mark --mark 0x0/0x10000 -j cali-from-hep-forward
-A cali-FORWARD -i cali+ -m comment --comment "cali:8ZoYfO5HKXWbB3pk" -j cali-from-wl-dispatch
-A cali-FORWARD -o cali+ -m comment --comment "cali:jdEuaPBe14V2hutn" -j cali-to-wl-dispatch
-A cali-FORWARD -m comment --comment "cali:12bc6HljsMKsmfr-" -j cali-to-hep-forward
-A cali-FORWARD -m comment --comment "cali:NOSxoaGx8OIstr1z" -j cali-cidr-block
-A cali-INPUT -p udp -m comment --comment "cali:w7ud0UgQSEi_zKuQ" -m comment --comment "Allow VXLAN packets from whitelisted hosts" -m multiport --dports 4789 -m set --match-set cali40all-vxlan-net src -m addrtype --dst-type LOCAL -j ACCEPT
-A cali-INPUT -p udp -m comment --comment "cali:4cgmbdWsLmozYhJh" -m comment --comment "Drop VXLAN packets from non-whitelisted hosts" -m multiport --dports 4789 -m addrtype --dst-type LOCAL -j DROP
-A cali-INPUT -i cali+ -m comment --comment "cali:t45BUBhpu3Wsmi1_" -g cali-wl-to-host
-A cali-INPUT -m comment --comment "cali:NOmsycyknYZaGOFf" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-INPUT -m comment --comment "cali:Or0B7eoenKO2p8Bf" -j MARK --set-xmark 0x0/0xf0000
-A cali-INPUT -m comment --comment "cali:AmIfvPGG2lYUK6mj" -j cali-from-host-endpoint
-A cali-INPUT -m comment --comment "cali:79fWWn1SpufdO7SE" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:Mq1_rAdXXH3YkrzW" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-OUTPUT -o cali+ -m comment --comment "cali:69FkRTJDvD5Vu6Vl" -j RETURN
-A cali-OUTPUT -p udp -m comment --comment "cali:R_U4Bsx2wPE7s0j6" -m comment --comment "Allow VXLAN packets to other whitelisted hosts" -m multiport --dports 4789 -m addrtype --src-type LOCAL -m set --match-set cali40all-vxlan-net dst -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:tQyQykplzpOIOeFr" -j MARK --set-xmark 0x0/0xf0000
-A cali-OUTPUT -m comment --comment "cali:teLcuBv_-wX6Mf5Z" -m conntrack ! --ctstate DNAT -j cali-to-host-endpoint
-A cali-OUTPUT -m comment --comment "cali:5wMpugAuGXHkWBaX" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-from-wl-dispatch -m comment --comment "cali:zTj6P0TIgYvgz-md" -m comment --comment "Unknown interface" -j DROP
-A cali-to-wl-dispatch -m comment --comment "cali:7KNphB1nNHw80nIO" -m comment --comment "Unknown interface" -j DROP
-A cali-wl-to-host -m comment --comment "cali:Ee9Sbo10IpVujdIY" -j cali-from-wl-dispatch
-A cali-wl-to-host -m comment --comment "cali:nSZbcOoG1xPONxb8" -m comment --comment "Configured DefaultEndpointToHostAction" -j ACCEPT

Not sure how to make it works, any insight would be greatly appreciated

godlzr avatar Oct 07 '21 20:10 godlzr

Looking at the logs.

Flannel is unable to start with this error.

Oct 07 20:05:35 canada-rustyspotted-com microk8s.daemon-flanneld[2877095]: Error:  client: endpoint https://10.0.0.57:12379 exceeded header timeout
Oct 07 20:05:35 canada-rustyspotted-com microk8s.daemon-flanneld[2877048]: /coreos.com/network/config is not in etcd. Probably a first time run.
Oct 07 20:05:36 canada-rustyspotted-com microk8s.daemon-flanneld[2877103]: Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 10.0.0.57:12379: connect: no route to host
Oct 07 20:05:36 canada-rustyspotted-com microk8s.daemon-flanneld[2877103]: error #0: dial tcp 10.0.0.57:12379: connect: no route to host
Oct 07 20:05:36 canada-rustyspotted-com systemd[1]: snap.microk8s.daemon-flanneld.service: Main process exited, code=exited, status=4/NOPERMISSION
Oct 07 20:05:36 canada-rustyspotted-com systemd[1]: snap.microk8s.daemon-flanneld.service: Failed with result 'exit-code'.
Oct 07 20:05:37 canada-rustyspotted-com systemd[1]: snap.microk8s.daemon-flanneld.service: Scheduled restart job, restart counter is at 3.
Oct 07 20:05:37 canada-rustyspotted-com systemd[1]: Stopped Service for snap application microk8s.daemon-flanneld.
Oct 07 20:05:37 canada-rustyspotted-com systemd[1]: Started Service for snap application microk8s.daemon-flanneld.

Is there a firewall that blocks the port 12379?

balchua avatar Oct 08 '21 02:10 balchua

@balchua thanks for looking into it, sorry but I am confused, the logs show the slave node can't access the master's etcd and the ip https://10.0.0.57:12379 is the local ip of the master node.

ip route
default via 10.0.0.1 dev enp0s3 proto dhcp src 10.0.0.57 metric 100 
10.0.0.0/24 dev enp0s3 proto kernel scope link src 10.0.0.57 
10.1.14.0/24 dev cni0 proto kernel scope link src 10.1.14.1 
10.1.27.0/24 via 10.1.27.0 dev flannel.1 onlink 
169.254.0.0/16 dev enp0s3 proto dhcp scope link src 10.0.0.57 metric 100

should the pod on the slave node communicate with the master node by internal ip? Did I misconfig the ip of the master node?

godlzr avatar Oct 13 '21 02:10 godlzr

Flannel is not deployed as a pod. It is a systemd process. Do you have firewall in between the nodes?

balchua avatar Oct 13 '21 04:10 balchua

I had a similar case where the worker's ClusterIP is not reachable on the controller machine using MicroK8s v1.23.6 and Ubuntu 22.04. However it works without issues on Ubuntu 20.04.

usersina avatar May 13 '22 10:05 usersina

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Apr 08 '23 12:04 stale[bot]