bridge-less aks-kubenet config

[eriksywu]

Our kubenet/route-based cni setup for containerd nodes can do without linux l2 bridges. Using a bridged kubenet plugin setup introduced a few painpoints:

1. hairpinMode is prone to device teardown bug in linux kernel (link here)
2. alternatively we use promiscMode which has its own issues, primarily the need to have a systemd service to register ebtables rules to drop duplicated packets from cbr0 which is on a very loose reconcile loop as a systemd service (link here) historically that was done as a part of the docker-shim code in kubelet
3. might be more prune to conntrack-induced latencies? (not currently backed by evidence)
4. ptp is another supported cni plugin that sets up veth pairs directly between a sandbox to the host instead of tying them to a linux bridge (cbr0) => thus making points 1-3 moot

unknowns:

• performance? i'm running cni perf tests in the bg
• does pod-pod traffic on the same host via service IP get hairpinned?

[paulgmiller]

Our cni plugins are pretty old. An update might effect perf results?

[alexeldeib]

@paulgmiller you might find this interesting