kilo
kilo copied to clipboard
Published
20 hours ago •
squat
squat
Network connectivity issue on RKE
Faced network connectivity issue between subnets on k8s nodes - k8s resources are reachable only within single node.
Initial setup: 2 node RKE k8s cluster. Nodes are placed in different availability zones, have only dedicated external IP addresses (no private networks attached, etc.).
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
$ docker version
Client: Docker Engine - Community
Version: 19.03.15
API version: 1.40
Go version: go1.13.15
Git commit: 99e3ed8919
Built: Sat Jan 30 03:16:51 2021
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.15
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 99e3ed8919
Built: Sat Jan 30 03:15:20 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.18.0
GitCommit: fec3683
This is how freshly installed k8s cluster looks like (without CNI plugins):
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
159.100.247.37 NotReady controlplane,etcd,worker 11m v1.19.6 159.100.247.37 <none> Ubuntu 18.04.5 LTS 4.15.0-136-generic docker://19.3.15
89.145.166.73 NotReady controlplane,etcd,worker 11m v1.19.6 89.145.166.73 <none> Ubuntu 18.04.5 LTS 4.15.0-136-generic docker://19.3.15
As there is no kilo manifest for RKE, I used kilo-k3s.yaml (it was also mentioned in one open issue here). After applying nodes become ready, coredns pods are up, node network interfaces are created, routes configured. Here are some outputs: node1:
# wg
interface: kilo0
public key: +SlpE7J0sq61ZJCbArlpYRS6BGYzy6qD0x+jZ618W08=
private key: (hidden)
listening port: 51820
peer: xX3yRb7vnH9h2mJ6PqCFUNWnosUjIzeR8KawQlxgym4=
endpoint: 159.100.247.37:51820
allowed ips: 10.42.0.0/24, 10.4.0.1/32
latest handshake: 14 seconds ago
transfer: 124 B received, 5.02 KiB sent
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 06:ed:94:00:08:e8 brd ff:ff:ff:ff:ff:ff
inet 89.145.166.73/23 brd 89.145.167.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::4ed:94ff:fe00:8e8/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:47:b4:4b:13 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:47ff:feb4:4b13/64 scope link
valid_lft forever preferred_lft forever
36: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.4.0.2/16 brd 10.4.255.255 scope global kilo0
valid_lft forever preferred_lft forever
37: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 2a:ae:38:3f:fa:1c brd ff:ff:ff:ff:ff:ff
inet 10.42.1.1/24 scope global kube-bridge
valid_lft forever preferred_lft forever
inet6 fe80::9cf6:93ff:feac:c751/64 scope link
valid_lft forever preferred_lft forever
40: vethd4c82b9f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master kube-bridge state UP group default
link/ether 32:73:51:9c:c8:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3073:51ff:fe9c:c813/64 scope link
valid_lft forever preferred_lft forever
41: vethb3e207e9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master kube-bridge state UP group default
link/ether 92:72:19:b0:ab:35 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::9072:19ff:feb0:ab35/64 scope link
valid_lft forever preferred_lft forever
42: vethc0e85852@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master kube-bridge state UP group default
link/ether 2a:ae:38:3f:fa:1c brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::28ae:38ff:fe3f:fa1c/64 scope link
valid_lft forever preferred_lft forever
# ip r
default via 89.145.166.1 dev eth0 proto dhcp src 89.145.166.73 metric 100
10.4.0.0/16 dev kilo0 proto kernel scope link src 10.4.0.2
10.42.0.0/24 via 10.4.0.1 dev kilo0 proto static onlink
10.42.1.0/24 dev kube-bridge proto kernel scope link src 10.42.1.1
89.145.166.0/23 dev eth0 proto kernel scope link src 89.145.166.73
89.145.166.1 dev eth0 proto dhcp scope link src 89.145.166.73 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
# cat /etc/cni/net.d/10-kilo.conflist
{"cniVersion":"0.3.1","name":"kilo","plugins":[{"bridge":"kube-bridge","forceAddress":true,"ipam":{"ranges":[[{"subnet":"10.42.1.0/24"}]],"type":"host-local"},"isDefaultGateway":true,"mtu":1450,"name":"kubernetes","type":"bridge"},{"capabilities":{"portMappings":true},"snat":true,"type":"portmap"}]}
node2:
# wg
interface: kilo0
public key: xX3yRb7vnH9h2mJ6PqCFUNWnosUjIzeR8KawQlxgym4=
private key: (hidden)
listening port: 51820
peer: +SlpE7J0sq61ZJCbArlpYRS6BGYzy6qD0x+jZ618W08=
endpoint: 89.145.166.73:51820
allowed ips: 10.42.1.0/24, 10.4.0.2/32
latest handshake: 16 seconds ago
transfer: 5.49 KiB received, 124 B sent
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 06:85:8c:00:08:76 brd ff:ff:ff:ff:ff:ff
inet 159.100.247.37/23 brd 159.100.247.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::485:8cff:fe00:876/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ef:9a:47:ec brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:efff:fe9a:47ec/64 scope link
valid_lft forever preferred_lft forever
36: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.4.0.1/16 brd 10.4.255.255 scope global kilo0
valid_lft forever preferred_lft forever
37: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 76:ce:99:2d:2c:04 brd ff:ff:ff:ff:ff:ff
inet 10.42.0.1/24 scope global kube-bridge
valid_lft forever preferred_lft forever
inet6 fe80::40e5:22ff:fe4d:355c/64 scope link
valid_lft forever preferred_lft forever
41: veth1e6e4c52@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master kube-bridge state UP group default
link/ether 76:ce:99:2d:2c:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::74ce:99ff:fe2d:2c04/64 scope link
valid_lft forever preferred_lft forever
42: veth6674dd51@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master kube-bridge state UP group default
link/ether ea:92:0a:4d:de:eb brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::e892:aff:fe4d:deeb/64 scope link
valid_lft forever preferred_lft forever
# ip r
default via 159.100.246.1 dev eth0 proto dhcp src 159.100.247.37 metric 100
10.4.0.0/16 dev kilo0 proto kernel scope link src 10.4.0.1
10.42.0.0/24 dev kube-bridge proto kernel scope link src 10.42.0.1
10.42.1.0/24 via 10.4.0.2 dev kilo0 proto static onlink
159.100.246.0/23 dev eth0 proto kernel scope link src 159.100.247.37
159.100.246.1 dev eth0 proto dhcp scope link src 159.100.247.37 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
# cat /etc/cni/net.d/10-kilo.conflist
{"cniVersion":"0.3.1","name":"kilo","plugins":[{"bridge":"kube-bridge","forceAddress":true,"ipam":{"ranges":[[{"subnet":"10.42.0.0/24"}]],"type":"host-local"},"isDefaultGateway":true,"mtu":1450,"name":"kubernetes","type":"bridge"},{"capabilities":{"portMappings":true},"snat":true,"type":"portmap"}]}
Pods:
$ kubectl get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx default-http-backend-65dd5949d9-gmdfn 1/1 Running 0 43m 10.42.1.4 89.145.166.73 <none> <none>
ingress-nginx nginx-ingress-controller-89d5q 1/1 Running 0 43m 159.100.247.37 159.100.247.37 <none> <none>
ingress-nginx nginx-ingress-controller-zphwr 1/1 Running 0 43m 89.145.166.73 89.145.166.73 <none> <none>
kube-system coredns-6f85d5fb88-sr9nw 1/1 Running 0 43m 10.42.0.4 159.100.247.37 <none> <none>
kube-system coredns-6f85d5fb88-zw2kg 1/1 Running 0 20m 10.42.1.5 89.145.166.73 <none> <none>
kube-system coredns-autoscaler-79599b9dc6-7nhc7 1/1 Running 0 43m 10.42.1.3 89.145.166.73 <none> <none>
kube-system kilo-5wqr5 1/1 Running 0 20m 89.145.166.73 89.145.166.73 <none> <none>
kube-system kilo-lp67n 1/1 Running 0 20m 159.100.247.37 159.100.247.37 <none> <none>
kube-system metrics-server-8449844bf-c4knp 1/1 Running 0 43m 10.42.0.3 159.100.247.37 <none> <none>
kube-system rke-coredns-addon-deploy-job-vlt5l 0/1 Completed 0 43m 89.145.166.73 89.145.166.73 <none> <none>
kube-system rke-ingress-controller-deploy-job-vffln 0/1 Completed 0 43m 89.145.166.73 89.145.166.73 <none> <none>
kube-system rke-metrics-addon-deploy-job-xz57c 0/1 Completed 0 43m 89.145.166.73 89.145.166.73 <none> <none>
kilo logs:
$ kubectl -n kube-system logs kilo-5wqr5
{"caller":"mesh.go:96","component":"kilo","level":"warn","msg":"no private key found on disk; generating one now","ts":"2021-03-03T08:29:06.92932142Z"}
{"caller":"main.go:221","msg":"Starting Kilo network mesh '2b959f7020a8dbb6b32860965ed4dbfd0dd11215'.","ts":"2021-03-03T08:29:06.941907053Z"}
{"caller":"cni.go:60","component":"kilo","err":"failed to read IPAM config from CNI config list file: invalid CIDR address: usePodCidr","level":"warn","msg":"failed to get CIDR from CNI file; overwriting it","ts":"2021-03-03T08:29:07.042344866Z"}
{"caller":"cni.go:68","component":"kilo","level":"info","msg":"CIDR in CNI file is empty","ts":"2021-03-03T08:29:07.042379524Z"}
{"CIDR":"10.42.1.0/24","caller":"cni.go:73","component":"kilo","level":"info","msg":"setting CIDR in CNI file","ts":"2021-03-03T08:29:07.042393076Z"}
E0303 08:29:07.103555 1 reflector.go:126] pkg/k8s/backend.go:407: Failed to list *v1alpha1.Peer: the server could not find the requested resource (get peers.kilo.squat.ai)
{"caller":"mesh.go:532","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-03-03T08:29:08.587825232Z"}
{"caller":"mesh.go:301","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"159.100.247.37","Port":51820},"Key":"eFgzeVJiN3ZuSDloMm1KNlBxQ0ZVTldub3NVakl6ZVI4S2F3UWx4Z3ltND0=","InternalIP":null,"LastSeen":1614760147,"Leader":false,"Location":"","Name":"159.100.247.37","PersistentKeepalive":0,"Subnet":{"IP":"10.42.0.0","Mask":"////AA=="},"WireGuardIP":null},"ts":"2021-03-03T08:29:08.616413782Z"}
{"caller":"mesh.go:532","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-03-03T08:29:08.678697039Z"}
{"caller":"mesh.go:301","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"159.100.247.37","Port":51820},"Key":"eFgzeVJiN3ZuSDloMm1KNlBxQ0ZVTldub3NVakl6ZVI4S2F3UWx4Z3ltND0=","InternalIP":null,"LastSeen":1614760147,"Leader":false,"Location":"","Name":"159.100.247.37","PersistentKeepalive":0,"Subnet":{"IP":"10.42.0.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="}},"ts":"2021-03-03T08:29:08.696034165Z"}
$ kubectl -n kube-system logs kilo-lp67n
{"caller":"mesh.go:96","component":"kilo","level":"warn","msg":"no private key found on disk; generating one now","ts":"2021-03-03T08:29:07.312678183Z"}
{"caller":"main.go:221","msg":"Starting Kilo network mesh '2b959f7020a8dbb6b32860965ed4dbfd0dd11215'.","ts":"2021-03-03T08:29:07.330771329Z"}
{"caller":"cni.go:60","component":"kilo","err":"failed to read IPAM config from CNI config list file: invalid CIDR address: usePodCidr","level":"warn","msg":"failed to get CIDR from CNI file; overwriting it","ts":"2021-03-03T08:29:07.431390019Z"}
{"caller":"cni.go:68","component":"kilo","level":"info","msg":"CIDR in CNI file is empty","ts":"2021-03-03T08:29:07.431433212Z"}
{"CIDR":"10.42.0.0/24","caller":"cni.go:73","component":"kilo","level":"info","msg":"setting CIDR in CNI file","ts":"2021-03-03T08:29:07.431446132Z"}
{"caller":"mesh.go:532","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-03-03T08:29:07.763066546Z"}
{"caller":"mesh.go:301","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"89.145.166.73","Port":51820},"Key":"K1NscEU3SjBzcTYxWkpDYkFybHBZUlM2QkdZenk2cUQweCtqWjYxOFcwOD0=","InternalIP":null,"LastSeen":1614760148,"Leader":false,"Location":"","Name":"89.145.166.73","PersistentKeepalive":0,"Subnet":{"IP":"10.42.1.0","Mask":"////AA=="},"WireGuardIP":null},"ts":"2021-03-03T08:29:08.324360744Z"}
{"caller":"mesh.go:532","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-03-03T08:29:08.428973858Z"}
{"caller":"mesh.go:301","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"89.145.166.73","Port":51820},"Key":"K1NscEU3SjBzcTYxWkpDYkFybHBZUlM2QkdZenk2cUQweCtqWjYxOFcwOD0=","InternalIP":null,"LastSeen":1614760148,"Leader":false,"Location":"","Name":"89.145.166.73","PersistentKeepalive":0,"Subnet":{"IP":"10.42.1.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.2","Mask":"//8AAA=="}},"ts":"2021-03-03T08:29:08.752187833Z"}
When I try to ping different pods from another pod or from the node itself - only pods which are hosted on the same node are reachable. Trying to ping coredns from node2:
root@node2:# ping 10.42.1.5
PING 10.42.1.5 (10.42.1.5) 56(84) bytes of data.
^C
--- 10.42.1.5 ping statistics ---
59 packets transmitted, 0 received, 100% packet loss, time 59371ms
root@node2:# # ping 10.42.0.4
PING 10.42.0.4 (10.42.0.4) 56(84) bytes of data.
64 bytes from 10.42.0.4: icmp_seq=1 ttl=64 time=0.041 ms
64 bytes from 10.42.0.4: icmp_seq=2 ttl=64 time=0.054 ms
64 bytes from 10.42.0.4: icmp_seq=3 ttl=64 time=0.039 ms
64 bytes from 10.42.0.4: icmp_seq=4 ttl=64 time=0.055 ms
64 bytes from 10.42.0.4: icmp_seq=5 ttl=64 time=0.051 ms
64 bytes from 10.42.0.4: icmp_seq=6 ttl=64 time=0.118 ms
64 bytes from 10.42.0.4: icmp_seq=7 ttl=64 time=0.086 ms
^C
--- 10.42.0.4 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6123ms
rtt min/avg/max/mdev = 0.039/0.063/0.118/0.027 ms
Thanks for the detailed report @3rmack. It occurs to me that there might be an issue with the default policy of the FORWARD chain on the filter table in iptables for your nodes. Can you please share the output of iptables-save?
node1:
# iptables-save
# Generated by iptables-save v1.6.1 on Thu Mar 4 07:24:07 2021
*mangle
:PREROUTING ACCEPT [295400:215490847]
:INPUT ACCEPT [293641:215410982]
:FORWARD ACCEPT [1759:79865]
:OUTPUT ACCEPT [292301:63812538]
:POSTROUTING ACCEPT [292296:63812238]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
COMMIT
# Completed on Thu Mar 4 07:24:07 2021
# Generated by iptables-save v1.6.1 on Thu Mar 4 07:24:07 2021
*filter
:INPUT ACCEPT [184600:35300734]
:FORWARD DROP [1752:79400]
:OUTPUT ACCEPT [188068:38373222]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu Mar 4 07:24:07 2021
# Generated by iptables-save v1.6.1 on Thu Mar 4 07:24:07 2021
*nat
:PREROUTING ACCEPT [2069:97392]
:INPUT ACCEPT [317:17992]
:OUTPUT ACCEPT [826:49560]
:POSTROUTING ACCEPT [826:49560]
:DOCKER - [0:0]
:KILO-NAT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-2AHCZLASALRZCLS7 - [0:0]
:KUBE-SEP-3YLRQZ6UJQXS6T5Q - [0:0]
:KUBE-SEP-6MIOBXLIHUQOTRKD - [0:0]
:KUBE-SEP-7AFK7QBGDHWZ2GEU - [0:0]
:KUBE-SEP-GQFJJFHKHZPIAODM - [0:0]
:KUBE-SEP-HOT2XMKPNFS7SQ2N - [0:0]
:KUBE-SEP-N53K6BQTZIOADP5D - [0:0]
:KUBE-SEP-PCJJFSWHCYBSYBAN - [0:0]
:KUBE-SEP-PJFTJTGBHOJ7TXPC - [0:0]
:KUBE-SEP-ZROOOGDTFBYPOKDJ - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-JTFAIQOSQRKTQWS3 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-QMWWTXBG7KFJQKLO - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.42.1.0/24 -m comment --comment "Kilo: jump to KILO-NAT chain" -j KILO-NAT
-A DOCKER -i docker0 -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.0.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.1.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -m comment --comment "Kilo: NAT remaining packets" -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE
-A KUBE-SEP-2AHCZLASALRZCLS7 -s 10.42.0.4/32 -m comment --comment "kube-system/metrics-server" -j KUBE-MARK-MASQ
-A KUBE-SEP-2AHCZLASALRZCLS7 -p tcp -m comment --comment "kube-system/metrics-server" -m tcp -j DNAT --to-destination 10.42.0.4:443
-A KUBE-SEP-3YLRQZ6UJQXS6T5Q -s 10.42.0.3/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-3YLRQZ6UJQXS6T5Q -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.42.0.3:9153
-A KUBE-SEP-6MIOBXLIHUQOTRKD -s 10.42.1.2/32 -m comment --comment "ingress-nginx/default-http-backend" -j KUBE-MARK-MASQ
-A KUBE-SEP-6MIOBXLIHUQOTRKD -p tcp -m comment --comment "ingress-nginx/default-http-backend" -m tcp -j DNAT --to-destination 10.42.1.2:8080
-A KUBE-SEP-7AFK7QBGDHWZ2GEU -s 10.42.1.3/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-7AFK7QBGDHWZ2GEU -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.42.1.3:9153
-A KUBE-SEP-GQFJJFHKHZPIAODM -s 10.42.0.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-GQFJJFHKHZPIAODM -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.42.0.3:53
-A KUBE-SEP-HOT2XMKPNFS7SQ2N -s 10.42.0.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-HOT2XMKPNFS7SQ2N -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.42.0.3:53
-A KUBE-SEP-N53K6BQTZIOADP5D -s 159.100.253.226/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-N53K6BQTZIOADP5D -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 159.100.253.226:6443
-A KUBE-SEP-PCJJFSWHCYBSYBAN -s 10.42.1.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-PCJJFSWHCYBSYBAN -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.42.1.3:53
-A KUBE-SEP-PJFTJTGBHOJ7TXPC -s 185.19.28.253/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-PJFTJTGBHOJ7TXPC -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 185.19.28.253:6443
-A KUBE-SEP-ZROOOGDTFBYPOKDJ -s 10.42.1.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZROOOGDTFBYPOKDJ -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.42.1.3:53
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.185.28/32 -p tcp -m comment --comment "kube-system/metrics-server cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.185.28/32 -p tcp -m comment --comment "kube-system/metrics-server cluster IP" -m tcp --dport 443 -j KUBE-SVC-QMWWTXBG7KFJQKLO
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.250.85/32 -p tcp -m comment --comment "ingress-nginx/default-http-backend cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.250.85/32 -p tcp -m comment --comment "ingress-nginx/default-http-backend cluster IP" -m tcp --dport 80 -j KUBE-SVC-JTFAIQOSQRKTQWS3
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-GQFJJFHKHZPIAODM
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-PCJJFSWHCYBSYBAN
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-3YLRQZ6UJQXS6T5Q
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-SEP-7AFK7QBGDHWZ2GEU
-A KUBE-SVC-JTFAIQOSQRKTQWS3 -m comment --comment "ingress-nginx/default-http-backend" -j KUBE-SEP-6MIOBXLIHUQOTRKD
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-N53K6BQTZIOADP5D
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-PJFTJTGBHOJ7TXPC
-A KUBE-SVC-QMWWTXBG7KFJQKLO -m comment --comment "kube-system/metrics-server" -j KUBE-SEP-2AHCZLASALRZCLS7
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-HOT2XMKPNFS7SQ2N
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-ZROOOGDTFBYPOKDJ
COMMIT
# Completed on Thu Mar 4 07:24:07 2021
node2:
# iptables-save
# Generated by iptables-save v1.6.1 on Thu Mar 4 07:27:11 2021
*mangle
:PREROUTING ACCEPT [245963:224655027]
:INPUT ACCEPT [240261:224344732]
:FORWARD ACCEPT [5702:310295]
:OUTPUT ACCEPT [236572:42704625]
:POSTROUTING ACCEPT [236567:42704325]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
COMMIT
# Completed on Thu Mar 4 07:27:11 2021
# Generated by iptables-save v1.6.1 on Thu Mar 4 07:27:11 2021
*filter
:INPUT ACCEPT [162605:29726290]
:FORWARD DROP [5616:305215]
:OUTPUT ACCEPT [163527:28386986]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu Mar 4 07:27:11 2021
# Generated by iptables-save v1.6.1 on Thu Mar 4 07:27:11 2021
*nat
:PREROUTING ACCEPT [5969:326387]
:INPUT ACCEPT [353:21172]
:OUTPUT ACCEPT [924:55536]
:POSTROUTING ACCEPT [924:55536]
:DOCKER - [0:0]
:KILO-NAT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-2AHCZLASALRZCLS7 - [0:0]
:KUBE-SEP-3YLRQZ6UJQXS6T5Q - [0:0]
:KUBE-SEP-6MIOBXLIHUQOTRKD - [0:0]
:KUBE-SEP-7AFK7QBGDHWZ2GEU - [0:0]
:KUBE-SEP-GQFJJFHKHZPIAODM - [0:0]
:KUBE-SEP-HOT2XMKPNFS7SQ2N - [0:0]
:KUBE-SEP-N53K6BQTZIOADP5D - [0:0]
:KUBE-SEP-PCJJFSWHCYBSYBAN - [0:0]
:KUBE-SEP-PJFTJTGBHOJ7TXPC - [0:0]
:KUBE-SEP-ZROOOGDTFBYPOKDJ - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-JTFAIQOSQRKTQWS3 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-QMWWTXBG7KFJQKLO - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.42.0.0/24 -m comment --comment "Kilo: jump to KILO-NAT chain" -j KILO-NAT
-A DOCKER -i docker0 -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.0.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.1.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -m comment --comment "Kilo: NAT remaining packets" -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE
-A KUBE-SEP-2AHCZLASALRZCLS7 -s 10.42.0.4/32 -m comment --comment "kube-system/metrics-server" -j KUBE-MARK-MASQ
-A KUBE-SEP-2AHCZLASALRZCLS7 -p tcp -m comment --comment "kube-system/metrics-server" -m tcp -j DNAT --to-destination 10.42.0.4:443
-A KUBE-SEP-3YLRQZ6UJQXS6T5Q -s 10.42.0.3/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-3YLRQZ6UJQXS6T5Q -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.42.0.3:9153
-A KUBE-SEP-6MIOBXLIHUQOTRKD -s 10.42.1.2/32 -m comment --comment "ingress-nginx/default-http-backend" -j KUBE-MARK-MASQ
-A KUBE-SEP-6MIOBXLIHUQOTRKD -p tcp -m comment --comment "ingress-nginx/default-http-backend" -m tcp -j DNAT --to-destination 10.42.1.2:8080
-A KUBE-SEP-7AFK7QBGDHWZ2GEU -s 10.42.1.3/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-7AFK7QBGDHWZ2GEU -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.42.1.3:9153
-A KUBE-SEP-GQFJJFHKHZPIAODM -s 10.42.0.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-GQFJJFHKHZPIAODM -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.42.0.3:53
-A KUBE-SEP-HOT2XMKPNFS7SQ2N -s 10.42.0.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-HOT2XMKPNFS7SQ2N -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.42.0.3:53
-A KUBE-SEP-N53K6BQTZIOADP5D -s 159.100.253.226/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-N53K6BQTZIOADP5D -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 159.100.253.226:6443
-A KUBE-SEP-PCJJFSWHCYBSYBAN -s 10.42.1.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-PCJJFSWHCYBSYBAN -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.42.1.3:53
-A KUBE-SEP-PJFTJTGBHOJ7TXPC -s 185.19.28.253/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-PJFTJTGBHOJ7TXPC -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 185.19.28.253:6443
-A KUBE-SEP-ZROOOGDTFBYPOKDJ -s 10.42.1.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZROOOGDTFBYPOKDJ -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.42.1.3:53
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.250.85/32 -p tcp -m comment --comment "ingress-nginx/default-http-backend cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.250.85/32 -p tcp -m comment --comment "ingress-nginx/default-http-backend cluster IP" -m tcp --dport 80 -j KUBE-SVC-JTFAIQOSQRKTQWS3
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.185.28/32 -p tcp -m comment --comment "kube-system/metrics-server cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.185.28/32 -p tcp -m comment --comment "kube-system/metrics-server cluster IP" -m tcp --dport 443 -j KUBE-SVC-QMWWTXBG7KFJQKLO
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-GQFJJFHKHZPIAODM
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-PCJJFSWHCYBSYBAN
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-3YLRQZ6UJQXS6T5Q
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-SEP-7AFK7QBGDHWZ2GEU
-A KUBE-SVC-JTFAIQOSQRKTQWS3 -m comment --comment "ingress-nginx/default-http-backend" -j KUBE-SEP-6MIOBXLIHUQOTRKD
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-N53K6BQTZIOADP5D
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-PJFTJTGBHOJ7TXPC
-A KUBE-SVC-QMWWTXBG7KFJQKLO -m comment --comment "kube-system/metrics-server" -j KUBE-SEP-2AHCZLASALRZCLS7
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-HOT2XMKPNFS7SQ2N
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-ZROOOGDTFBYPOKDJ
COMMIT
# Completed on Thu Mar 4 07:27:11 2021
@3rmack were you able to uncover the source of this issue? From what I can see, the iptables rules look totally fine. To begin testing, we could check:
- can you ping the Kilo interface from another node? I.e.
ping 10.4.0.1andping 10.4.0.2 - if this works, then we would want ping the coredns pod on node2 from node1 and tcpdump the kilo0 interface on node2; do we see and ICMP packets coming from that interface?