bridge: ARP entry not populated when reusing pod IP.

[DrmagicE]

Hi, teams. Recently, we encountered a weird problem in our newly set up Kubernetes cluster.

When creating a new pod, sometimes the pod network took a relatively long time to take effect (about 10 seconds), leading to connection timeout problems.

We found that all problematic pods were reusing previous pod's IP. (Our cluster podCIDR is small.)

It seems the mac address of the new pod is not populated in the ARP table:

# 1. The pod with mac 22:47:9a:ce:70:88 is deleted.
# 2. The new pod reusing the same IP 10.244.0.13 is created, but we still get the old mac address.
[root@localhost ~]# ip n | grep 10.244.0.13
10.244.0.13 dev cni0 lladdr 22:47:9a:ce:70:88 STALE
[root@localhost ~]# ping 10.244.0.13
PING 10.244.0.13 (10.244.0.13) 56(84) bytes of data.
64 bytes from 10.244.0.13: icmp_seq=10 ttl=64 time=0.210 ms
64 bytes from 10.244.0.13: icmp_seq=11 ttl=64 time=0.116 ms
64 bytes from 10.244.0.13: icmp_seq=12 ttl=64 time=0.090 ms
^C
--- 10.244.0.13 ping statistics ---
12 packets transmitted, 3 received, 75% packet loss, time 11018ms
rtt min/avg/max/mdev = 0.090/0.138/0.210/0.053 ms
# After the slow ping, the host can get the right mac address for the new pod.
[root@localhost ~]# ip n | grep 10.244.0.13
10.244.0.13 dev cni0 lladdr 2e:db:fb:ac:d1:d6 REACHABLE

The issue gets fixed after downgrading the bridge plugin back to 0.9.1.

Here are some steps and scripts to help to reproduce the problem:

1. Prepare a single node Kubernetes Cluster with /28 podCIDR.
2. Create a bunch of test pods:

for i in {1..10}; do   kubectl run --image nginx:latest nginx-$i; done

3. Once all pods get ready, ping all pod ips from host:

kubectl get pods -ojsonpath='{.items[*].status.podIP}' | xargs -n1 ping -c 2

4. Delete all test pods:

for i in {1..10}; do   kubectl delete pod nginx-$i; done

5. Repeat step 2 and 3, and you will find some ping stuck for about 10 seconds .

Related issue: #756

Environment information: Kubernetes: v1.25.6 CNI: v1.2.0 Kernel: 3.10.0-1160.el7.x86_64 OS: CentOS 7.6