tailscale
tailscale copied to clipboard
tailscale
macOS: When using an Exit Node and `--accept-dns=false` all system DNS resolution is broken because no resolver is set
What is the issue?
Per #12231 I am running Tailscale with --accept-dns=false as a workaround for a bug in Tailscale not updating my local DNS resolver.
However, I've discovered that with --accept-dns=false now Exit Nodes are effectively broken. It seems that even when MagicDNS is off Tailscale will setup 100.100.100.100 as a resolver when using an Exit Node, however turning off accept-dns prevents this. As a result no resolver is set on the system! This can be observed by seeing that /etc/resolv.conf is empty.
Steps to reproduce
- Be on macOS
- Run
tailscale set --accept-dns=falseto prevent Tailscale from setting any system DNS settings. - Switch to using an exit node
- Observe that all system DNS resolution is broken. Try running
ping www.google.com. - Run
cat /etc/resolv.confto observe that no nameserver is set. - Now run
tailscale set --accept-dns=trueto observer that now a nameserver is set and DNS resolution is working again.
The above behavior seems identical whether MagicDNS is enabled or not.
Possibly related to my other DNS+macOS bug report #12231, at least in how its related to Tailscale taking over all system DNS.
Are there any recent changes that introduced the issue?
I am a new Tailscale user
OS
macOS
OS version
14.3.1
Tailscale version
1.66.4 tailscale commit: 067defc641850791142cc83a44dadae29709344a
Other software
I'm running Little Snitch but all the testing done above was done with Little Snitch off. (however it was still installed on the system, so that might be contributing).
I also ensured that iCloud Private Relay was off (both system wide and the Wi-Fi network specific "Limit IP Address tracking" option).
Bug report
BUG-cafb7d6238f79c6992e84aac8608faa66cc8d12fb844958915c46f6c20dd3592-20240525193733Z-1d4fc7fafa5df705
Investigating this.
Note that /etc/resolv.conf is not always the right place to identify what DNS resolver a macOS system is using. Prefer using scutil --dns, which provides more information including scoped resolvers, such as those used in a split DNS configuration.
We are experiencing this issue on macosx sonoma 14.5. We are trying to roll out tailscale across the org and this will prevent further testing of the product.
I've noticed when enabling exit node while tailscale dns settings are disabled the resulting /etc/resolv.conf is empty which is effectively causing DNS to not work. We prefer to not use the tailscale dns settings.
To note -- scutil --dns does show items there but it does not appear to actually be using these settings. Additionally, turning off exit nodes and running the same command shows the exact same details but the difference is the /etc/resolv.conf is populated with previous values.
This was working with a previous version of tailscaled on the linux system, we just redeployed the exit node and now we're getting this issue.
@gyoza can you clarify what you mean by we just redeployed the exit node, are you saying this stopped working as a result of upgrading the Tailscale client on the node that was serving as exit node?
I can confirm this issue exists and I can reproduce. The issue stems from having an exit node enabled (which instructs Tailscale to ask macOS to send all routes through the tunnel) and at the same time having no DNS server defined for the VPN interface. We're looking into a sensible fix that doesn't require admins to manually configure the DNS server to be used in this scenario.
@gyoza can you clarify what you mean by
we just redeployed the exit node, are you saying this stopped working as a result of upgrading the Tailscale client on the node that was serving as exit node?
This is correct. The exit node was working fine this morning and we wanted to get more network performance so we bumped up the instance type.
The ubuntu exit node was possibly running 1.66.3 but i cant say for sure. (Forgive me because I did not look at the version before rebuilding.)
The version now is 1.66.4
We have another system that is identical running on Ubuntu 1.66.3 and I set it to advertise as an exit node and that does not appear to fix it. Same symptoms persist. Thanks ahead of time for looking at this, you guys are awesome.
Yeah, I was surprised to hear this would be related to a client upgrade on the exit node. The problem appears to be entirely in our macOS client. It's not easy to solve. I could tell the client to look up the DNS server used by your external network interface, and use that whenever, but this would break any setup where a DNS configuration profile is being used.
Temporary workaround
To unblock you while I come up with something better, it appears possible to use scutil to explicitly set the DNS servers to be used by the VPN interface. Run the following after picking an exit node in the Tailscale menu:
$ sudo scutil
> list
The above will print a list of services, find the UUID of the Tailscale service (it should be contained in the only line that ends in VPN, if Tailscale is the only VPN installed), for instance: subKey [155] = State:/Network/Service/DB5586F7-6051-4364-BDC7-0E7F18C0A323/VPN
Then, do the following to set the DNS servers for the Tailscale service:
$ sudo scutil
> open
> d.init
> d.add ServerAddresses * 8.8.8.8 9.9.9.9
> set State:/Network/Service/TAILSCALE_SERVICE_UUID/DNS
> quit
scutil --dns should now list 8.8.8.8 and 9.9.9.9 as the resolvers used for the utun interface.
Currently, one way to solve this bug would be to add an option to let admins configure these fallback DNS servers with a single CLI command or MDM-deployable setting. For instance:
defaults write io.tailscale.ipn.macos FallbackDNSServers -array 8.8.8.8 9.9.9.9
But I'd rather spend some extra time on it and come up with something that works out of the box if possible.
Note that
/etc/resolv.confis not always the right place to identify what DNS resolver a macOS system is using. Prefer usingscutil --dns
The issue is that scutil --dns is showing a resolver that apparently isn't being used and only /etc/resolv.conf hints that something is amiss. But now I realize that I should be looking for a resolver on the Tailscale VPN interface.
Thanks for this awesome workaround solution using scutil. I love learning more about macOS networking internals. To make it easier, here's a version of your solution with a little more automation to make it require no interaction. (Finding the UUID was trickier for me since I have many other VPN interfaces).
# Get the Tailscale service UUID
TS_UUID="$(scutil --nc list | awk '/io.tailscale.ipn.macos/ {print $3}')"
# pipe the scutil interactive commands in directly
sudo scutil <<EOF
open
d.init
d.add ServerAddresses * 1.1.1.1
set State:/Network/Service/$TS_UUID/DNS
quit
EOF
edit: I've kept having to use this workaround often so I created a little shell function for it: https://gist.github.com/varenc/971796720a31c3b16e6c99903794bc1e
Status update: we have built and are currently verifying/testing a new UI pane in the DNS settings view to specify custom DNS (or DoH!) servers to be used for tunnel-bound traffic when Tailscale DNS is disabled. The settings will also be remotely configurable via MDM. I expect this new feature will land in an unstable build some time early next week.
Hoping for this fix! Any update? I've been using this shell script to automate manually adding a resolver to the Tailscale network service. While it works well it's still a little tedious to remember to call it every time I start using an exit node.
(for context I keep --accept-dns=false because I use dnscrypt-proxy for DNS privacy/security, and because I have other various resolver rules to handle some special cases. Including rules to that route *.ts.net and 100.64.0.0/10 to the Tailscale 100.100.100.100 resolver.)
Hoping for this fix! Any update? I've been using this shell script to automate manually adding a resolver to the Tailscale network service. While it works well it's still a little tedious to remember to call it every time I start using an exit node.
(for context I keep
--accept-dns=falsebecause I usednscrypt-proxyfor DNS privacy/security, and because I have other various resolver rules to handle some special cases. Including rules to that route *.ts.net and 100.64.0.0/10 to the Tailscale 100.100.100.100 resolver.)
It's coming into unstable, sometime this week or the next one. We had to spend some time thinking about the UI and implications of the change.
Any update on when we can expect this in a stable release? Wondering if it makes sense to fix it with the manual steps described in the comments or to wait for a release.
Thank you!
Facing the same issue here, Set Remote VPS/VDS as Exit Node. but the device using the exit node have no Internet.
Waiting for this "specify custom DNS (or DoH!" especially for set designate custom nameservers for designate Exit Node! https://github.com/tailscale/tailscale/issues/13705#issuecomment-2401131090
root@device:~# tailscale set --accept-dns=false
root@device:~# cat /etc/resolv.conf
# Tailscale DNS
nameserver 100.100.100.100
search tailnetname.ts.net
nameserver 1.1.1.1
nameserver 8.8.8.8
root@device:~# nano /etc/resolv.conf
root@device:~# chattr -i /etc/resolv.conf
root@device:~# nano /etc/resolv.conf
root@device:~# chattr +i /etc/resolv.conf
root@device:~# ping google.com
PING google.com(nchkga-ao-in-x0e.1e100.net (2404:6800:4005:825::200e)) 56 data bytes
64 bytes from nchkga-ao-in-x0e.1e100.net (2404:6800:4005:825::200e): icmp_seq=1 ttl=117 time=5.23 ms
64 bytes from nchkga-ao-in-x0e.1e100.net (2404:6800:4005:825::200e): icmp_seq=2 ttl=117 time=4.47 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 4.467/4.850/5.233/0.383 ms
root@device:~# systemctl restart tailscaled
root@device:~# dig @100.100.100.100 google.com
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> @100.100.100.100 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 7983
;; flags: qr aa rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;google.com. IN A
;; Query time: 0 msec
;; SERVER: 100.100.100.100#53(100.100.100.100) (UDP)
;; WHEN: Sat Nov 23 10:28:51 CST 2024
;; MSG SIZE rcvd: 28
I'm happy to confirm this change works great! And the default 'Automatic' Advanced DNS Setting works exactly like how I would expect. When I switch to using an exit node, Tailscale sets its network service DNS server, to whatever my primary interface's DNS server was set to. Basically it keeps my DNS server the same which is exactly what I want. Thanks for being awesome Tailscale team!
one minor complaint: When using Automatic Advanced DNS while on an Exit Node, Tailscale doesn't update its DNS to reflect changes in the DNS server used by my primary network interface. For example: I I'm on Ethernet using one DNS server and activate an Exit Node, Tailscale uses that Ethernet adapter's DNS settings, which is great! But if I then unplug from Ethernet so that now Wi-Fi is my primary interface, Tailscale doesn't update its DNS to reflect whatever my Wi-Fi interface's DNS is.
This would only matter if the DNS server I'm using via Ethernet isn't accessible over Wi-Fi since I'm on my neighbor's Wi-Fi network or something. Similarly if I change my Ethernet adapter's DNS server while using an Exit Node, Tailscale doesn't react to this. (Like if I run networksetup -setdnsservers Ethernet <new_dns_server> it has no effect).
The situations in my life where I'm doing this are pretty rare though and just toggling the exit node on/off is an easy fix. This bug already only affected a small number of users and my minor complaint affects even fewer. I think it's plenty good currently and for the %0.01 of users that also notice this we can just toggle the Exit Node.
Cheers!
Yeah, the current implementation doesn't react to network changes. We certainly need to do so, and we'll look into adding this to a future release.
in Tailscale v1.78. remote VPC in Debian 12; have tailscale set --accept-dns=false, and config the systemd-resolved via "/etc/systemd/resolved.conf". doubled checked there is no NetworkManager installed or running. Read carefully this manual again, The Sisyphean Task Of DNS Client Config on Linux, still can't using exit node. with --accept-dns=true/false. or in iOS tailscale DNS Setting config "Advanced DNS Settings to Automatic or Custom with '1.1.1.1'
chattr -i /etc/resolv.conf
rm /etc/resolv.conf
sudo apt install systemd-resolved
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
sudo systemctl enable systemd-resolved.service
sudo systemctl start systemd-resolved.service
add below to /etc/systemd/resolved.conf
DNS=100.100.100.100
FallbackDNS=1.1.1.1 8.8.8.8
Domains=tailname.ts.net
If keep the tailscale accept dns false; could ping through like "ping -c 5 mac-mini"
root@device:~# cat /etc/systemd/resolved.conf
[Resolve]
DNS=100.100.100.100
FallbackDNS=1.1.1.1 8.8.8.8
Domains=tailname.ts.net
root@device:~# cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
nameserver 127.0.0.53
options edns0 trust-ad
search tailname.ts.net
If enable the tailscale accept dns true, tailscaled will override /etc/resolv.conf
root@device:~# cat /etc/resolv.conf
# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
nameserver 100.100.100.100
search tailname.ts.net tailname.ts.net
Both case can't using exit node. shows like DNS querying ? In previous version before tailscale v1.78. it just show error as there is only "100.100.100.100" DNS nameserver. "resolvectl status" shows below.
root@device:~# nano /etc/systemd/resolved.conf
root@device:~# sudo systemctl restart systemd-resolved
root@device:~# resolvectl status
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
DNS Servers 100.100.100.100
Fallback DNS Servers 1.1.1.1 8.8.8.8
DNS Domain tailname.ts.net
Link 2 (ens3)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 1.1.1.1 8.8.8.8
Link 3 (tailscale0)
Current Scopes: LLMNR/IPv6
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 4 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
the above config still show error in tailscale status sometimes "- Tailscale can't reach the configured DNS servers. Internet connectivity may be affected."