Connection to K8S Service - SourceIP is not preserved (Source NAT)

[ch9hn]

Hello together, we are using a managed K8S cluster with the following Kilo image: docker.io/squat/kilo:bcb722b0b90271a6643d6360e69e702d578d6a06

What we have done so far: Added a Kilo peer via the following manifest:

apiVersion: kilo.squat.ai/v1alpha1
kind: Peer
metadata:
  name: gw-peer
spec:
  allowedIPs:
    - 10.4.0.99/32
    - 192.168.0.0/24 # Test Client Network (
  publicKey: <PUBKEY>
  persistentKeepalive: 10

On the side of the VPN peer we have the following config, which was generated by kgctl:

[Interface]
Address = 10.4.0.99/24
SaveConfig = true
PrivateKey = <PRIVATEKEY>

[Peer]
PublicKey = <PUBKEY>
AllowedIPs = 100.64.2.0/24, 10.4.0.1/32
Endpoint = 51.15.70.238:51820
PersistentKeepalive = 10

[Peer]
PublicKey = <PUBKEY>
AllowedIPs = 100.64.4.0/24, 10.4.0.2/32, 10.32.0.0/12 # Service CIDR is added here.
Endpoint = 51.15.100.213:51820
PersistentKeepalive = 10

[Peer]
PublicKey = <PUBKEY>
AllowedIPs = 100.64.3.0/24, 10.4.0.3/32
Endpoint = 51.158.232.221:51820
PersistentKeepalive = 10

[Peer]
PublicKey = <PUBKEY>
AllowedIPs = 100.64.1.0/24, 10.4.0.4/32
Endpoint = 51.158.241.235:51820
PersistentKeepalive = 10

Now to the issue:

When we are sending UDP packets to our pod, the client IP needs to be preserved. This is no problem, when we are sending on one Pod IP like 100.64.4.4.

But when we are sending the UDP packets to a K8S Service like this:

apiVersion: v1
kind: Service
metadata:
  name: svc-connector
  namespace: <MYSPACE>
spec:
  type: ClusterIP
  clusterIP: 10.40.40.40 # We set the clusterIP field, to identify and hardcode the IP.
  ports:
    - name: connector-udp
      protocol: UDP
      port: 5684
      targetPort: 5684
  selector:
    app: <MYAPP>
    component: test-connector

Then the Source IP is not preserved and we are seeing the IP address of the node.

UDP server message sent to 100.64.4.1:59778
processed data for xxxxx from 100.64.4.1`

We need to preserve the source IP.

How we can do this? Which options we need to switch on ?

[squat]

Hi @chfxr I did some digging and found the cause of this SNAT. Kilo masquerades packets when they come from peers [0] and go to IPs that Kilo doesn't know about. In other words, Kilo does not masquerade packets that are destined for known IPs [1]. The main goal here is to ensure that packets that are leaving the cluster can find their way back to private IPs under control of Kilo, like the IPs of peers. Today, service IPs are IPs that Kilo doesn't know about! This is rather something under the control of kube-proxy and similar projects. Nevertheless, service IPs are internal to the cluster and the pods that ultimately receive the packet would be able to respond. Furthermore, if it's internal to the cluster, then this masquerading doesn't provide any benefit and only adds overhead, so I would argue that Kilo should allow packets destined for service IPs to go through without being masqueraded.

The main challenge is that Kilo doesn't know the Service CIDR for the cluster. One solution that occurs to me is to make the preservation of source IPs an opt-in feature that is enabled if you deploy Kilo with a --service-cidr flag. We could then maybe use this value for additional features. Would an opt-in solution like this work for you?

[0] https://github.com/squat/kilo/blob/main/pkg/mesh/routes.go#L368 [1] https://github.com/squat/kilo/blob/main/pkg/mesh/routes.go#L352

[ch9hn]

Hello @squat, thank you very much for your explanation. The Service CIDR range of a cluster can be determined via the following script:

SVCRANGE=$(echo '{"apiVersion":"v1","kind":"Service","metadata":{"name":"tst"},"spec":{"clusterIP":"1.1.1.1","ports":[{"port":443}]}}' | kubectl apply -f - 2>&1 | sed 's/.*valid IPs is //') echo $SVCRANGE Therefore the range is known then and can obtained and loaded into Kilo.

A --service-cidr flag would be sufficient to use this feature.

[squat]

A --service-cidr flag would be sufficient to use this feature.

Excellent I'll put something together.

[gtriggiano]

Hi @squat. First thing first let me thank you for your awesome work :)

Are there any news about this?

I'm using Kilo on top of Cilium to allow my team to have a VPN access to cluster services. Of course we would not want to provide indiscriminate access to all services and we'd like to leverage CiliumNetworkPolicies to segment traffic.

However without this feature it would be useless to enumerate peers IP in network policies ingress rules, because all traffic to services backing pods will appear as coming from the cilium host. Moreover, in CNP, we need to allow ingress from host nodes which we also would like to avoid.

[squat]

hi @gtriggiano sorry I forgot about this issue! I just pushed a PR that implements the suggested --service-cidr flag: #351. Please take a look. I'll merge this soon so that you and @ch9hn can give it a test drive.

[gtriggiano]

Thank you very much @squat! Looking forward for the merged feature. Do you plan to craft a new versioned release soon?

[squat]

@gtriggiano @ch9hn this feature is in Kilo 0.6.0 🎉🎉