Kube OVN external gateway config namespace

[insignia96]

I have been working on using some Kube OVN features in my cluster and I was finally able to get the EIP and SNAT features working in the default VPC.

I had to make the following patches to the kube-ovn controller and daemonset in order to make this work. In the kube-ovn-controller deployment and the kube-ovn-cni DaemonSet --external-gateway-config-ns=cozy-kubeovn needs to be added, as the service account used cannot access the ConfigMap when it is placed in the default kube-system namespace.

Looking at the upstream Helm chart, it doesn't seem to provide a way to configure these values at all. Either way, I just figured I would put this out there for anyone attempting to do the same type of thing. My use case is for outbound SMTP from a tenant cluster, and the need for a consistent egress IP.

[kvaps]

Hey, many thanks for your research. We have implemented stateless 1:1 NAT using cozy-proxy controller. It works outside of Kube-OVN. The only thing you need to do is to add:

networking.cozystack.io/wholeIP: "true"

annotation to your service.

This design was chosen because we want to follow native Kubernetes networking pattern and expose VMs through services with a single IPAM.

Please try out of it is solves your problem.

Here is an example for VMs: https://github.com/cozystack/cozystack/blob/31b110cd39eabd6cf47381f4b487620ffffce50f/packages/apps/vm-instance/templates/service.yaml#L11

[insignia96]

This makes perfect sense to me. I hadn't tried using the WholeIP feature in this way, and I didn't realize it was able to handle egress traffic. I have experimented with using a modified version of the Kubernetes helm chart to deploy node groups with the SNAT annotation. I suppose with this method you could do the same thing. Allocate a pool of SMTP nodes and a MetalLB pool with sufficient space for the maximum replica count, annotate those nodes to receive addresses from the special pool and use WholeIP, then you know all nodes with that role will have a static external IP from the SMTP pool.

I will try this out and make sure it works. Adding the annotation to the mail service in the tenant cluster and having it pass up to the parent cluster service didn't seem to affect the egress traffic from the tenant cluster node or nodes. It makes sense to me why, but I figured it was worth a try lol.

[insignia96]

After some more tinkering, this does work, but since the WholeIP feature is 1:1, I haven't yet figured out a clean way to handle a node group with autoscaling, and creating the corresponding services on the fly. Creating a service which matches on the node group results in only the first VM getting exposed and then only that node has its egress traffic rewritten to the service IP.

[dosubot[bot]]

Hi, @insignia96. I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

Issue Summary

• You reported the need to patch the Kube OVN controller and daemonset to add the --external-gateway-config-ns flag for proper ConfigMap access, as the upstream Helm chart does not support this.
• The maintainer suggested using the cozy-proxy controller's WholeIP feature with a specific annotation to enable stateless 1:1 NAT and expose VMs via services.
• You confirmed that this approach works well for static IP assignment.
• However, you noted challenges with autoscaling node groups, where only the first VM gets exposed and egress rewritten.
• Further exploration is needed to handle dynamic scaling scenarios cleanly.

Next Steps

• Please let me know if this issue is still relevant with the latest version of the cozystack repository by commenting here to keep the discussion open.
• Otherwise, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!