Hybrid-NAT failed, EDGE-MESH chain is not in the right order, cloud pod can not access edge service.

[yoseihouyn]

What happened: Restart k8s node, sometimes EDGE-MESH chain is after KUBE-SERVICES chain(edgemesh-agent is started before kube-proxy), cloud pod can not access edge service:

iptables -t nat -L PREROUTING

Chain PREROUTING (policy ACCEPT) target prot opt source destination
cali-PREROUTING all -- anywhere anywhere /* cali:6gwbT8clXdHdC1b1 / KUBE-SERVICES all -- anywhere anywhere / kubernetes service portals / EDGE-MESH all -- anywhere anywhere / edgemesh root chain / DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL TUNNEL-PORT all -- anywhere anywhere / kubeedge tunnel port */

iptables -t nat -L OUTPUT

Chain OUTPUT (policy ACCEPT) target prot opt source destination
cali-OUTPUT all -- anywhere anywhere /* cali:tVnHkvAo15HuiPy0 / KUBE-SERVICES all -- anywhere anywhere / kubernetes service portals / EDGE-MESH all -- anywhere anywhere / edgemesh root chain / DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL TUNNEL-PORT all -- anywhere anywhere / kubeedge tunnel port */

kubectl get pods -n kubeedge -o wide

What you expected to happen: cloud pod can access edge service after restart k8s cloud node.

How to reproduce it (as minimally and precisely as possible): Restart k8s cloud node.

Anything else we need to know?:

Environment:

• EdgeMesh version: 1.9.0

• Kubernetes version (use kubectl version): v1.21.4

• KubeEdge version(e.g. cloudcore --version and edgecore --version):cloudcore v1.8.2, edgecore v1.8.1

• Cloud nodes Environment:

  • Hardware configuration (e.g. lscpu): Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz
  • OS (e.g. cat /etc/os-release): CentOS Linux release 7.9.2009 (Core)
  • Kernel (e.g. uname -a):3.10.0-1160.42.2.el7.x86_64
  • Go version (e.g. go version):
  • Others:
• Edge nodes Environment:

  • edgecore version (e.g. edgecore --version):
  • Hardware configuration (e.g. lscpu):
  • OS (e.g. cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Go version (e.g. go version):
  • Others:

[Poorunga]

Currently, you can only keep the EDGE-MESH chain in front of KUBE-SERVICES by deleting the EDGE-MESH chain and then recreating the edgemesh.

$ iptables -t nat -D OUTPUT 3
$ iptables -t nat -D PREROUTING 3

3 means the EDGE-MESH chain is in 3rd order.

[Poorunga]

When the KUBE-SERVICES chain exists, kube-proxy will not rebuild the KUBE-SERVICES chain. Rebuilding the KUBE-SERVICES chain will insert the first place in the system root chain (such as OUTPUT, PREROUTING). edgemesh also uses the same logic as kube-proxy to handle EDGE-MESH chains. Let's make an assumption: if kube-proxy and edgemesh rebuild their own chains and insert them into the first one every time they are started or restarted, it will cause competition between the two, resulting in system instability. This is clearly a bad behavior.

---

当KUBE-SERVICES链存在的时候，kube-proxy不会重建KUBE-SERVICES链，重建KUBE-SERVICES链会在系统根链（比如 OUTPUT, PREROUTING）的首位插入。edgemesh也采用了和kube-proxy相同的逻辑处理EDGE-MESH链。 我们做一个假设：如果kube-proxy和edgemesh每次启动或者重启就重建自己的链并插到第一条，那么就会导致两者出现竞争，导致系统不稳定。这显然是一个不好的行为。

[yoseihouyn]

As per my understanding, Maybe we can fix this issue by using init-container to check the readiness of kube-proxy container. In the cloud node, maybe we can use the following comand to check: $(curl -k -o /dev/null -s -w %{http_code} https://10.96.0.1:443/api/v1/namespaces/default/services/kubernetes) == 200 if kube-proxy and cni plugin is running, it will be success.

Another way is check the order of EDGE-MESH chain every second...

[yoseihouyn]

As per my understanding, Maybe we can fix this issue by using init-container to check the readiness of kube-proxy container. In the cloud node, maybe we can use the following comand to check: $(curl -k -o /dev/null -s -w %{http_code} https://10.96.0.1:443/api/v1/namespaces/default/services/kubernetes) == 200 if kube-proxy and cni plugin is running, it will be success.

Another way is check the order of EDGE-MESH chain every second...

calico CNI plugin use the second method to keep cali-PREROUTING chain always in the 1st order.

[Poorunga]

I am more worried about whether edgemesh will always have various problems when inserting the EDGE-MESH chain into the 1st order. However, it is fine to ensure that EDGE-MESH is placed in front of the KUBE-SERVICES chain.

According to the method you proposed above, it is possible to ensure that EDGE-MESH is placed in front of the KUBE-SERVICES chain. Are you willing to solve this problem?

[yoseihouyn]

Updated 05-daemonset.yaml file in the helm chart, and now it works fine... The code as below:

But this is just a temporary solution, the better solution should be the second one....

[yoseihouyn]

I am more worried about whether edgemesh will always have various problems when inserting the EDGE-MESH chain into the 1st order. However, it is fine to ensure that EDGE-MESH is placed in front of the KUBE-SERVICES chain.

According to the method you proposed above, it is possible to ensure that EDGE-MESH is placed in front of the KUBE-SERVICES chain. Are you willing to solve this problem?

Let me find how to contribute codes to this project....