netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Magic DNS doesn't work "The host dns manager does not support match domains"

Open TheLinuxGuy opened this issue 5 months ago • 9 comments

Describe the problem

After a reboot, Netbird DNS does not work immediately, requiring manual troubleshooting... it seems to "fix itself" sometimes.

System is DietPi.com based on Debian 12 bookworm.

To Reproduce

Steps to reproduce the behavior:

  1. Have a working netbird peer host. Reboot it.
  2. Upon reboot, check netbird status -d it will be missing Name Servers and showing The host dns manager does not support match domains

Expected behavior

NetBird Magic DNS should just work after a system reboot.

Are you using NetBird Cloud?

Yes

NetBird version

0.49.0

Is any other VPN software installed?

No

Debug output

To help us resolve the problem, please attach the following anonymized status output

netbird status -dA <- attached .txt

netbird-status.txt

Create and upload a debug bundle, and share the returned file key:

f79e391890ab27fb37c88b3b4be7011e22aa2e5ca6f38ffa9c4481884941f726/30b20564-53e0-4c05-be6e-31b110301109

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

wt0 should not be missing the netbird magic dns entry below.

root@NEXUS:~# uptime
 00:16:40 up 0 min,  1 user,  load average: 0.08, 0.02, 0.01
root@NEXUS:~# resolvectl status
Global
       Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
      DNS Servers 8.8.8.8

Link 2 (eth0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wt0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

A manual restart sometimes fixes it.

root@NEXUS:~# netbird down
Disconnected
root@NEXUS:~# netbird up
Connected
root@NEXUS:~# resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
Current DNS Server: 8.8.8.8
        DNS Servers 8.8.8.8

...

Link 14 (wt0)
    Current Scopes: DNS
         Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.122.137.151
       DNS Servers: 100.122.137.151
        DNS Domain: in.mesh ~122.100.in-addr.arpa
root@NEXUS:~#

TheLinuxGuy avatar Jun 27 '25 04:06 TheLinuxGuy

If it doesn't work to begin with, but later it kicks in I think it either started before systemd-resolved, or there is some logic flaw in the DNS manager detection at https://github.com/netbirdio/netbird/blob/52ff9d960213181a49cfa5f6bdfae6fb9bf3f513/client/internal/dns/host_unix.go#L77-L123

Could you post the content of /etc/resolv.conf and determine whether it changes during boot?

nazarewk avatar Jun 27 '25 07:06 nazarewk

Sure. I am going to cat the file with uptime, then reboot and do the same thing.

root@gw:~# uptime; cat /etc/resolv.conf
 12:06:08 up 10:31,  1 user,  load average: 0.00, 0.00, 0.00

nameserver 127.0.0.53
options edns0 trust-ad
search in.mesh

On fresh reboot, the file did change to 127.0.0.53 - don't know why.

root@gw:~# uptime; cat /etc/resolv.conf
 12:07:04 up 0 min,  1 user,  load average: 1.04, 0.22, 0.07

nameserver 127.0.0.53
options edns0 trust-ad
search .

TheLinuxGuy avatar Jun 27 '25 16:06 TheLinuxGuy

@TheLinuxGuy 127.0.0.53 will be systemd-resolved's local resolver, which is correct, but I am not seeing the usual comments coming with the systemd-resolved, this is how it looks on my system:

> cat -n /etc/resolv.conf 
     1  # This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
     2  # Do not edit.
     3  #
     4  # This file might be symlinked as /etc/resolv.conf. If you're looking at
     5  # /etc/resolv.conf and seeing this text, you have followed the symlink.
     6  #
     7  # This is a dynamic resolv.conf file for connecting local clients to the
     8  # internal DNS stub resolver of systemd-resolved. This file lists all
     9  # configured search domains.
    10  #
    11  # Run "resolvectl status" to see details about the uplink DNS servers
    12  # currently in use.
    13  #
    14  # Third party programs should typically not access this file directly, but only
    15  # through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
    16  # different way, replace this symlink by a static file or a different symlink.
    17  #
    18  # See man:systemd-resolved.service(8) for details about the supported modes of
    19  # operation for /etc/resolv.conf.
    20  
    21  nameserver 127.0.0.53
    22  options edns0 trust-ad
    23  search ... netbird.selfhosted  ....

The current detection is searching for a systemd-resolved string inside of /etc/resolv.conf and this is where the current detection fails:

https://github.com/netbirdio/netbird/blob/52ff9d960213181a49cfa5f6bdfae6fb9bf3f513/client/internal/dns/host_unix.go#L103-L109

nazarewk avatar Jun 27 '25 16:06 nazarewk

Did you configure anything in particular about your OS installation? Could you also provide ls -la /etc/resolv.conf & resolvectl status?

Might help us adjust the resolver discovery process.

nazarewk avatar Jun 27 '25 16:06 nazarewk

Did you configure anything in particular about your OS installation? Could you also provide ls -la /etc/resolv.conf & resolvectl status?

Yes, fresh dietpi.com install on VM/VPS and then installed essential netbird required packages prior to netbird install, the following commands were ran:

  • apt-get install systemd-resolved dbus
  • systemctl enable dbus
  • systemctl start dbus
  • systemctl enable systemd-resolved.service --now
root@gw:/etc/komodo# ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jun 27 01:31 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
root@gw:/etc/komodo# resolvectl status
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
Current DNS Server: 1.1.1.1
        DNS Servers 1.1.1.1

Link 2 (eth0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (wt0)
    Current Scopes: DNS
         Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.122.228.183
       DNS Servers: 100.122.228.183
        DNS Domain: in.mesh ~122.100.in-addr.arpa
root@gw:/etc/komodo# 
root@gw:/etc/komodo# cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search in.mesh

I am not seeing the usual comments coming @nazarewk sorry about that... I trimmed the usual stuff. Its there.

TheLinuxGuy avatar Jun 27 '25 16:06 TheLinuxGuy

If you have those in place. I suspect a startup ordering issue on the system if you're only having NetBird fail during the boot process and succeeding after the restart of the netbird service.

Could you show systemctl status dbus systemd-resolved netbird right after rebooting?

nazarewk avatar Jun 27 '25 17:06 nazarewk

root@gw:~# systemctl status dbus systemd-resolved netbird
● dbus.service - D-Bus System Message Bus
     Loaded: loaded (/lib/systemd/system/dbus.service; static)
     Active: active (running) since Fri 2025-06-27 13:22:16 EDT; 4s ago
TriggeredBy: ● dbus.socket
       Docs: man:dbus-daemon(1)
   Main PID: 515 (dbus-daemon)
      Tasks: 1 (limit: 2274)
     Memory: 2.1M
        CPU: 33ms
     CGroup: /system.slice/dbus.service
             └─515 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

Jun 27 13:22:16 gw systemd[1]: Starting dbus.service - D-Bus System Message Bus...
Jun 27 13:22:16 gw systemd[1]: Started dbus.service - D-Bus System Message Bus.

● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-06-27 13:22:16 EDT; 4s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 318 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 2274)
     Memory: 7.0M
        CPU: 193ms
     CGroup: /system.slice/systemd-resolved.service
             └─318 /lib/systemd/systemd-resolved

Jun 27 13:22:15 gw systemd-resolved[318]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 27 13:22:15 gw systemd-resolved[318]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16
Jun 27 13:22:15 gw systemd-resolved[318]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa corp home internal intranet lan local private test
Jun 27 13:22:16 gw systemd-resolved[318]: Using system hostname 'gw'.
Jun 27 13:22:16 gw systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set DNS server list to: 100.122.228.183
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set DNSSEC setting: no
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set default route setting: no
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set search domain list to: in.mesh., ~122.100.in-addr.arpa.
Jun 27 13:22:18 gw systemd-resolved[318]: Flushed all caches.

● netbird.service - Netbird mesh network client
     Loaded: loaded (/etc/systemd/system/netbird.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-06-27 13:22:16 EDT; 3s ago
   Main PID: 606 (netbird)
      Tasks: 7 (limit: 2274)
     Memory: 56.1M
        CPU: 356ms
     CGroup: /system.slice/netbird.service
             └─606 /usr/bin/netbird service run --config /etc/netbird/config.json --log-level info --daemon-addr unix:///var/run/netbird.sock --log-file /var/log/netbird/client.log

Jun 27 13:22:16 gw systemd[1]: Started netbird.service - Netbird mesh network client.
root@gw:~# date
Fri Jun 27 13:22:22 EDT 2025
root@gw:~# uptime
 13:22:24 up 0 min,  1 user,  load average: 0.22, 0.05, 0.02
root@gw:~# 

TheLinuxGuy avatar Jun 27 '25 17:06 TheLinuxGuy

seems like it worked this time for you according to the systemd-resolved?

Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set DNS server list to: 100.122.228.183
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set DNSSEC setting: no
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set default route setting: no
Jun 27 13:22:18 gw systemd-resolved[318]: wt0: Bus client set search domain list to: in.mesh., ~122.100.in-addr.arpa.

nazarewk avatar Jun 27 '25 17:06 nazarewk

seems like it worked this time for you according to the systemd-resolved?

Yes, on this instance it recovered almost immediately. Not to say that is the behavior I was seeing before filing this bug though.

TheLinuxGuy avatar Jun 27 '25 18:06 TheLinuxGuy

Not sure if it's related but I have this error on a machine that's running pi-hole (a local DNS resolver) that binds to port 53. systemd-resolved, however, binds its stub listener to that port so for pi-hole to be able to use it it's necessary to reconfigure it like this:

/etc/systemd/resolved.conf.d/disable-stub-listener.conf

[Resolve]
DNSStubListener=no

I don't know of a way to have Netbird's automatic DNS resolution working with this setup but I can script it by parsing the output of netbird status -d to grab the client mappings out of it. For example, to get /etc/hosts style mappings, you could run:

netbird status --json | jq -r '.peers.details.[] | [.netbirdIp, .fqdn] | @tsv'

andrejcremoznik avatar Jul 23 '25 06:07 andrejcremoznik

@andrejcremoznik thanks for the tip for generating the hosts file.

It is currently impossible to implement it any other way than manually registering entries and/or pointing the specific domain to NetBird's local resolver in your upstream resolver (PiHole instance in your case) periodically. On some specific DNS managers (eg: resolved) this is simply automated.

The default internal NetBird resolver runs at port 53 of NetBird's IP by default:

root@brys-vm-nbt-ubuntu-01:~# ss -nlp 'sport 53'
Netid      State        Recv-Q       Send-Q              Local Address:Port             Peer Address:Port      Process                                         
udp        UNCONN       0            0                  100.83.128.237:53                    0.0.0.0:*          users:(("netbird",pid=3392,fd=32))   

You can also tell it to run at a custom IP:port instead with $NB_DNS_RESOLVER_ADDRESS or --dns-resolver-address flag on netbird up. This is how I'm doing it on my NixOS:

kdn@brys ~> cat $(which netbird-priv)
#! /nix/store/00zrahbb32nzawrmv9sjxn36h7qk9vrs-bash-5.2p37/bin/bash -e
...
export NB_DNS_RESOLVER_ADDRESS=${NB_DNS_RESOLVER_ADDRESS-'127.5.18.19:53'}
...
exec "/nix/store/xykz2jlyrhngdnhidyq9bn749sp4rkqc-netbird-0.49.0/bin/netbird"  "$@" 

For CoreDNS a forward to the resolver's IP with netbird.cloud should suffice, I used to do this like this, except resolving netbird.cloud with systemd-resolved instead of directly from NetBird resolver.

nazarewk avatar Jul 23 '25 08:07 nazarewk