Proper KubeSpan port forwarding support

[stevefan1999-personal]

I want to experiment Talos between my home and my Azure network over IPv4, and my home network is obviously having a NAT with a private network range behind a router. More specifically, I want to have my homelab 1 and homelab 2, hidden behind my home network as 192.168.0.1 and 192.168.0.2 respectively, to be able to connect to my Azure machine as 10.10.10.1. So, having a VPN, or rather an overlay network is on the right track and KubeSpan is clearly the choice here.

However, far as I can tell, KubeSpan actually requires public IP addresses for the peers to discover each other (which to be honest, covers 90% of the cases for Kubernetes deployment), and so while it is likely that Wireguard can run as a one-armed bandit if one side has a public IP and the other side doesn't (but that means you would have a unidirectional channel which could slow things down), but if both sides doesn't, no dice.

Adding salt to injury, most home network routers only support symmetric NAT (you can understand it as iptables SNAT/Masquerade if you want to make things easier), so the source port can be rewritten at router level when sending out. Therefore, a proper NAT traversal solution is needed for KubeSpan, similar to the kind of Tailscale or NetBird, and use TURN/ICE as a last resort.

Originally posted by @stevefan1999-personal in https://github.com/siderolabs/talos/discussions/8338

We should be more cautious on assumption the network environment of each user. As for me, I have multiple Kubernetes cluster running in my homelab and I really cannot modify the DMZ settings.

So I have to filter my home settings to avoid the LAN IP addresses:

    kubespan:
      enabled: true # Enable the KubeSpan feature.
      harvestExtraEndpoints: true
      allowDownPeerBypass: true
      filters:
        endpoints:
        - '0.0.0.0/0'
        - '!10.0.0.0/8'
        - '!10.240.0.0/12'
        - '!10.96.0.0/12'
        - '!2001:db8:42:0::/56'
        - '!2001:db8:42:1::/112'
        - '!192.168.2.0/24'
        - '!fd81:25b4:68e1::/48'

But then that ended up:

139.xxx.xxx.85    kubespan    KubeSpanPeerSpec   E+EuruO95kq/vpDuwZlQJQovf3iboDo/f1O4YEqzgRk=   1         talos-gbu-8kf       []

So KubeSpan clearly failed to take in this case, and failed to resolve the IP address for the control plane.

For other uses, I have a bunch of server behind CGNAT which means ingress/egress port forwarding is also effective unavailable, as that means the endpoint IP address from that CGNAT peer has to always be assumed as dynamic.

As such I think we need to setup proper port forwarding support, using the right tech such as STUN or TURN. At the very least, need to get some mechanism to let any incoming address as endpoints (aka passive WireGuard tunnel):

    kubespan:
      enabled: true # Enable the KubeSpan feature.
      harvestExtraEndpoints: true
      allowDownPeerBypass: true
      dynamic: true
      filters:
        endpoints:
        - '0.0.0.0/0'
        - '!10.0.0.0/8'
        - '!10.240.0.0/12'
        - '!10.96.0.0/12'
        - '!2001:db8:42:0::/56'
        - '!2001:db8:42:1::/112'
        - '!2001:470:19:c5d::/64'
        - '!192.168.2.0/24'
        - '!fd81:25b4:68e1::/48'