tailscale
Tailsacle accepted subnet routes should not not have a higher priority than main table
What is the issue?
In a simple Tailscale setup, we have two nodes. The node 1 advertises some routes and the node 2 accepts them:
Tailscale accepts subnet routes from other nodes and adds them to the routing table. But Tailscale is not adding them to the main table, it's adding them to the table 52 and adding a rule to let the traffic lookup table 52 firstly and then the main table:
https://github.com/tailscale/tailscale/blob/74637f2c156a91d24d429f44dd0f0b888cd7a4f2/wgengine/router/router_linux.go#L1039-L1047
- I think it's dangerous since it introduces the risk to lose remote access from local network (outgoing traffic even belonging to connection from local network is routed to Tailscale).2.
- It breaks static routes -- e.g. static routes configured in network management tools (e.g. systemd-networkd) are added to main table with a customizable metric -- but I can't enforce which one has higher priority by setting the static routes' metric. The only way is adding
ip rules manully and adding static routes to another separated route table -- messing up the rules and routes.
In my opinion, as a overlay network, Tailscale should not trying to override the physical network and break current configuration on the installed router/host. It should use the general practice for solving routing conflict in network systems -- route metric, to give the control to the user, not forcing the priority by ip rules.
Steps to reproduce
No response
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
No response
Tailscale version
1.32.2
Bug report
No response
The subnet routes are added to table 52 to avoid the risk of a local network, like a public Wi-Fi network, using the same IP addresses as an exported subnet route. So for example accessing a subnet route to an office network using 10.0.0.0/24 at the same time that one is on a public Wi-Fi network also using 10.0.0.0/24.
The protection should be done by dropping forward packets from unwanted interfaces to Tailscale interfaces with iptables.
Other related issues: #5999 #1344 #1227
I appreciate the goal of Tailscale being "dead simple", but it terrifies me that there might be some future "magic" added that could cause significant semantics changes to networking. I guess it only impacts me when I'm in transition to a full tailscale deployment, but transitions can take a long time...
I understand that for an unsophisticated end user in a coffee shop, that is trying to access a tailscale-advertised 192.168.0.0/24 network, things become difficult if you don't have this network override. However, on the other hand having a tailscale-advertised 10.51.0.0/16 route to your office networks block office machines access to their local 10.51.1.0/24 LAN is surprising and unexpected. To someone familiar with networking, breaking the "a more specific route overrides a less specific one" is unexpected.
This is also slightly confusing because the subnet routers (in this case, the machine advertising 10.51.0.0/16) must be a special case, where the networking works normally (because otherwise it would just loop in tailscale). So that node is able to access 10.51.1.0/24 normally.
I would propose a couple potential options:
- Make this functionality a switch: "Coffee Shop Routing Override" or something? :-)
- Alert the user when they are in this situation ("I see you are on a local network that matches a tailnet advertised network."). That means that the user then needs to think and react.
- Only take over the route if the prefix length matches? IOW: A Tailscale 10.51.1.0/24 route could take over for a 10.51.1.0/24 local network, but a 10.51.0.0/16 route would not. It would still be surprising, but at least then you have conflicting routes.
My Use Case
I'm trying to migrate a small company over to Tailscale as an overlay network. This includes both server to server communications, and workstation or laptop use. I would like to do this incrementally, setting up users on an existing firewall to reach internal resources, adding all servers to the tailscale, eventually moving potentially all communications over to Tailscale, but before then having all machines on the tailnet, while still allowing existing communications to happen.
I have ~10 users with workstations and other devices, 3-4 locations, and ~5 networks at each location (dev, staging, DMZ, private, etc). Each location has a 10.X/16 subnet, and each network has a 10.X.Y/24 subnet. Each location has an existing firewall and VPN concentrator.
So my thinking is that I'll start the transition by setting up my workstations and the firewalls with tailscale, have the firewalls advertise their 10.X/16 location networks, and then use my legacy firewall rules to allow me to access the existing resources using existing rules. Then I'll start adding tailscale to the individual nodes in the locations so I can start transitioning over to tailscale ACLs and direct access to resources. Seems fairly straightforward, but I added tailscale to a handful of servers and all hell breaks loose because they can no longer access other servers on their LAN, unless I use "--accept-routes=false".
Workarounds:
- On the machines on the tailscale advertised network, use "--accept-routes=false". This is workable in my case because the majority of my servers will not be using firewall routed networks (the exception being monitoring servers and a build server), and workstations or laptops can probably go through the firewall even if they are directly connected to one of these networks (I'm sure there will be special cases I run into, particularly for workstations in the dev network, where my more advanced users will beat their head against "why can't I access port 8080 of this test machine").
- (Linux) Delete the subnet route from table 52: ip ro del table 52 10.51.0.0/16
- (Linux) Add a higher-priority rule to route the subnets via the local table: ip rule add to 10.51.1.0/24 priority 5000 table main
Those latter two might be options you want to add an to an "After" clause to the tailscale service ("systemctl edit tailscaled"), though I'd worry about it not being a true "post script" and there being cases where it either runs before tailscale does it's thing, or if the tailnet bounces that the After doesn't catch future instances where it would need to run.
My workaround is set TS_DEBUG_USE_IP_COMMAND=1 to forcibly make tailscaled to use ip command instead of Linux APIs. And provide a fake ip command (by appending $PATH) which doesn't support ip rule. Then tailscaled will only add routes to the main table.
# cat /opt/tailscale-patch/bin/ip
#!/bin/bash
if [[ "$1" == "rule" ]]; then
exit 1
fi
for DIR in /bin /sbin /usr/bin /usr/sbin; do
[[ -f "$DIR/ip" ]] && exec "$DIR/ip" "$@"
done
exit 2
# cat /etc/default/tailscaled
# Set the port to listen on for incoming VPN packets.
# Remote nodes will automatically be informed about the new port number,
# but you might want to configure this in order to set external firewall
# settings.
PORT="41641"
# Extra flags you might want to pass to tailscaled.
FLAGS=""
TS_DEBUG_MTU="1400"
TS_DEBUG_USE_IP_COMMAND="1"
# cat /etc/systemd/system/tailscaled.service.d/override.conf
[Service]
Environment=PATH=/opt/tailscale-patch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
The subnet routes are added to table 52 to avoid the risk of a local network, like a public Wi-Fi network, using the same IP addresses as an exported subnet route. So for example accessing a subnet route to an office network using 10.0.0.0/24 at the same time that one is on a public Wi-Fi network also using 10.0.0.0/24.
@DentonGentry I don't think you're understanding the problem. While there may be some concern about remote, portable machines, the concern expressed here is for devices which DO want to communicate on their attached LAN. The entire documented subnet router model does not work if the same unit accepts routes from another client that also advertises. A subnet router that can't communicate on the LAN is pretty worthless.
If we need redundancy in the subnet routers, it doesn't work.
If we want to put nodes on the same lan as a subnet router, it doesn't work for those nodes.
This bug report has nothing to do remote nodes, on which a user can make choices. Every person who has posted in this thread is talking about core infrastructure, not remote access to overlapping subnets.
There are two intertwined issues:
- for tailnets which wish to allow it, permit nodes to communicate directly on the LAN without going through the advertised route from a subnet router.
- not allow a coffee shop wifi using 192.168.1.0/24 to inadvertently take over the advertised 192.168.1.0/24 route.
I expect to handle this with functionality in tailscaled to determine if it is on the same LAN segment as the other tailnet node it is attempting to communicate with, by exchanging packets containing a nonce and trying to determine whether they can connect directly.
There are two intertwined issues:
- for tailnets which wish to allow it, permit nodes to communicate directly on the LAN without going through the advertised route from a subnet router.
- not allow a coffee shop wifi using 192.168.1.0/24 to inadvertently take over the advertised 192.168.1.0/24 route.
Not in this topic. To date Tailscale has solved one only by breaking the other.
I expect to handle this with functionality in tailscaled to determine if it is on the same LAN segment as the other tailnet node it is attempting to communicate with, by exchanging packets containing a nonce and trying to determine whether they can connect directly.
It would be great to see a solution, any solution. So far it's been 27 months without even a proposal to fix it, and no answer to the dozens of reports of the problems it creates.
That being said, your proposed solution involves considerable complexity in comparison to what I proposed in #7947. Nothing in your solution seems to address more complex networks, where more than one route should be routed through a local router.
When will this be fixed? I don't even see any conclusion, not to mention the plan. Seem like tailscale team just ignore this, while I believe this must be a huge problem for almost any use-cases.
Is there a workaround for the problem? I want to use 10.192.0.20/28 from my OpenVPN interface, but I also have a tailscale subnet router with 10.0.0.0/8. The practice of longest prefix match no longer applies here. Is it perhaps possible to install a special tailscale subnet with a lower priority?
The same thing also happens without accept-routes if I have only activated one exit node. The only workaround for me at the moment is to use classic WireGuard to the subnet router and to the exit node.
@marek22k you could solve this by using a higher (better) priority for your OpenVPN routes. See https://superuser.com/a/1224642/528643
You could add the relevant ip-rule command to route-up.sh to make sure your routes get higher priority, e.g.
/bin/ip rule add from all lookup 100 priority 99
The priority (99) just needs to be higher (better) than the tailscale priority of 5270:
...
5270: from all lookup 52
I do exactly this with my other native wireguard connections so they take preference over tailscale (and wg-quick has a nice Table config option to automatically use a different routing table).
Because I always want my higher-priority route table to exist on boot, on Ubuntu I use a systemd service template and enable the service on boot:
/etc/systemd/system/[email protected]:
[Unit]
Description=IP rule for %I (%i)
# Usage: systemctl {start|enable} ip-rule@{TABLE}-{PRIORITY}.service
# e.g. systemctl start [email protected]
# where 20 is the table number, 99 is the priority
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/bash -c 'X="%i"; TABLE=$${X%-*}; PRIO=$${X#*-}; /bin/ip rule add from all lookup $$TABLE priority $$PRIO'
ExecStop=/usr/bin/bash -c 'X="%i"; TABLE=$${X%-*}; PRIO=$${X#*-}; /bin/ip rule del from all lookup $$TABLE priority $$PRIO'
[Install]
WantedBy=multi-user.target
Then enable with:
systemctl enable [email protected]
systemctl start [email protected]
and you'll always have table 100 at priority 99 after each boot.
This is still not a good idea as it is a Linux specific solution to a problem affecting all operating systems. Instead of putting all routes into a separate routing table routes should be added with a user configurable priority. That way it should work on anything based on BSD routing -- even BSD (doh!). Anything else will bite you further down the way.
