pkg/guestagent/iptables: recognize Kubernetes NodePorts

[AkihiroSuda]

I guess we need to adjust the regexp here to support kube NodePorts ?

https://github.com/lima-vm/lima/blob/4edd35381c48e33721f54868d75408a4e6b41c8c/pkg/guestagent/iptables/iptables.go#L20-L34

Originally posted by @AkihiroSuda in https://github.com/lima-vm/lima/discussions/1018#discussioncomment-3405760

[ilya1st]

copy last comment here @AkihiroSuda may be. I use kubernetes 1.24 and in uses nftables for forwarding not iptables In nft ruleset it looks like:

iptables cannot see nft rules

[afbjorklund]

For the k8s example it seems to work, they are shown with the usual iptables ?

anders@lima-k8s:~$ sudo kubectl get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-minikube1   NodePort    10.103.113.80   <none>        8080:31605/TCP   73s
kubernetes        ClusterIP   10.96.0.1       <none>        443/TCP          6h2m
anders@lima-k8s:~$ sudo iptables -t nat -S | grep 31605
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/hello-minikube1" -m tcp --dport 31605 -j KUBE-EXT-IUOMQFLPVEEG3DF5

Simple echoserver as per https://minikube.sigs.k8s.io/docs/handbook/accessing/

---

One would have to dig a bit to get to the actual DNAT, since it uses the Cluster IP.

-A KUBE-NODEPORTS -p tcp -m comment --comment "default/hello-minikube1" -m tcp --dport 31605 -j KUBE-EXT-IUOMQFLPVEEG3DF5 -A KUBE-EXT-IUOMQFLPVEEG3DF5 -j KUBE-SVC-IUOMQFLPVEEG3DF5 -A KUBE-SVC-IUOMQFLPVEEG3DF5 ! -s 10.244.0.0/16 -d 10.103.113.80/32 -p tcp -m comment --comment "default/hello-minikube1 cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ -A KUBE-SEP-KONAZYSBK4GHAOZH -p tcp -m comment --comment "default/hello-minikube1" -m tcp -j DNAT --to-destination 10.244.0.13:8080

iptables v1.8.7 (nf_tables)

[jandubois]

FWIW, we are planning to implement a component for Rancher Desktop to use the apiserver endpoint to watch for service port changes, and then forward them accordingly. This feels more robust than updating parsing rules for iptables and nftables to reverse engineer the modified ports.

This will only work for k8s, so the existing iptables parsing needs to stay to handle containerd port usage without k8s.

We can put this code into lima as well (once it exists), but I'm not sure if lima should include code talking to the apiserver (I think that will be fine, but it would be an extension of scope).

[AkihiroSuda]

We can put this code into lima as well (once it exists),

SGTM

[jandubois]

We can put this code into lima as well (once it exists),

Looks like it was already written while I was on vacation: https://github.com/rancher-sandbox/rancher-desktop-agent/tree/main/pkg/kube 😄

It just creates a corresponding listener on 127.0.0.1, so the regular lima port forwarding will pick it up (same thing kubernetes used to do before).

Still need to figure out how it should be integrated into Lima.

[afbjorklund]

This feels more robust than updating parsing rules for iptables and nftables to reverse engineer the modified ports.

Upstream kubernetes seems to agree on that, too: https://kubernetes.io/blog/2022/09/07/iptables-chains-not-api/

[saltbo]

Has the plan been decided? maybe I can work for it

[saltbo]

Temporary solution: exec nc -l localhost 31313 in lima-vm

[saltbo]

iptables -t nat -vnL KUBE-NODEPORTS| awk '/dpt/ {print $NF}' | cut -c 5-9

Can we watch the iptables to open a listener for the nodePort?