Feature Request: Add custom routes outside the nebula's unsafe_routes (for K8S Flannel)

[xiaolei0125]

I run the docker container on multiple nebula nodes, Containers on each node have different networks, such as 172.16.12.0/24, 172.16.21.0/24;

I try to add custom routes in Linux route tables like this: route add -net 172.16.12.0 netmask 255.255.255.0 gw 192.168.10.12 route add -net 172.16.21.0 netmask 255.255.255.0 gw 192.168.10.21

But it doesn't work, I can't ping container's ip on other Nebula nodes; I have to specify the routes for each container network in nebula unsafe_routes, like this; unsafe_routes: - route: 172.16.12.0/24 via: 192.168.10.12 - route: 172.16.21.0/24 via: 192.168.10.21

Use unsafe_routes, can ping container's ip on other Nebula nodes.

When we use K8S Flannel to manage container networks, the network is dynamic allocation by K8S Flannel, not fixed. Flannel can manage the routing of container network by itself, and add the corresponding route to the host.

So the problem is that Flannel can not work with Nebula, and other app like Flannel which need route also can not work with Nebula；

It's seem Nebula will queryUnsafeRoute to determine which ip can routable, so if add custom routes outside the nebula's unsafe_routes, Nebula will drop it;

I know many P2P VPN tools can support to add custom routes in host route table, such as ZeroTier, Tinc VPN and WireGuard, can Nebula also support this ?

[nbrownus]

This could as simple as making via optional and only installing the routes when via is present. Will try to get it in for the 1.6 release.

[nbrownus]

I started playing with this over here https://github.com/nbrownus/nebula/tree/system-routing feedback greatly appreciated!

[tarrenj]

I started playing with this over here https://github.com/nbrownus/nebula/tree/system-routing feedback greatly appreciated!

Have you been able to get a successful test with your patch yet, or are you still working on that?

[nbrownus]

Have you been able to get a successful test with your patch yet, or are you still working on that?

I have tested that the route changes get consumed by nebula but I haven't tested beyond that at this point. I believe it should work for folks.

[kaplan-michael]

I should be able to test it a bit tomorrow, I'll report back how it went.

[kaplan-michael]

Hey @nbrownus, took a little longer to get to it. I can confirm the routes are picked up by nebula, but traffic can't actually flow through them. level=debug msg="dropping outbound packet, vpnIp not in our CIDR or in unsafe_routes ..."

Am I missing some config, or could it be that it just doesn't recognize the destination IP so it drops it?

[yaroslav-gwit]

I am very interested in this feature too. Nebula as a mesh VPN + OSPF would be a dream come true, because at the moment I have to go through a manual process of updating my Ansible inventory file(s) to populate the unsafe routes to all my nodes. I wanted to open a new request for this feature, but it's great that I am not alone with such an idea.

On the other hand, this should not be a default behaviour, because it might be exploited in the future. One could probably implement use_system_routing_table: true/false config flag to harden the setup.