netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Published 20 hours ago • verified publisher icon netbirdio

DNS Search Domains have stopped working

Open drewhemm opened this issue 5 months ago • 4 comments
trafficstars

Describe the problem

Domains configured in Netbird are no longer usable for searching by hostname, including netbird.selfhosted.

All hosts must now be queried using their fully-qualified domain names.

To Reproduce

Steps to reproduce the behavior:

  1. Configure a Nameserver in Netbird
  2. Assign it to one or more distribution groups
  3. Add a match domain
  4. Select the option for 'Mark match domains as search domains'
  5. Connect a Netbird peer to the mesh
  6. Run netbird status -d to confirm that the nameserver is 'Available' and that the domain name is listed in the output
  7. Try to resolve a hostname, such as peer.example.com using only peer - it will fail

Expected behavior

peer.example.com should be resolvable using nslookup peer.

peer2.netbird.selfhosted should be resolvable using nslookup peer2

Are you using NetBird Cloud?

Self-hosted

NetBird version

0.45.1

Is any other VPN software installed?

No

Debug output

To help us resolve the problem, please attach the following anonymized status output

Peers detail:
 netbird2-a2sdv-1.netbird.selfhosted:
  NetBird IP: 100.71.49.170
  Public key: MNDLBqPRHHd1HdbGnIIXQKP5aZA9B7dDw0vSxd7DXAI=
  Status: Connecting
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 43 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s
 netbird-bmg-aws-uk-sbx-audiobroadcast.netbird.selfhosted:
  NetBird IP: 100.71.56.47
  Public key: aQepL1EcWM4JMjrqb6bPVwJT6aFGCFEh3zAIHlWVZiI=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): relay/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:55166/198.51.100.1:16206
  Relay server address:
  Last connection update: 26 seconds ago
  Last WireGuard handshake: 26 seconds ago
  Transfer status (received/sent) 252 B/308 B
  Quantum resistance: false
  Networks: 10.231.230.0/23
  Latency: 16.106ms
 rpi4b-window.netbird.selfhosted:
  NetBird IP: 100.71.117.141
  Public key: wAKpG4Ol+aSzF9wAlhFwFYLxrIwQT3qKSndxVVUscEE=
  Status: Connecting
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 43 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s
 netbird-dev-optimiser-k8s-78dfdb66c6-lm98v.netbird.selfhosted:
  NetBird IP: 100.71.162.200
  Public key: toQZNEV4BOJtvJrDXubRwSwfLoQImc3IPtu7syPxcQ4=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:50694/198.51.100.2:1024
  Relay server address:
  Last connection update: 26 seconds ago
  Last WireGuard handshake: 26 seconds ago
  Transfer status (received/sent) 412 B/404 B
  Quantum resistance: false
  Networks: 10.243.0.0/22, 10.243.4.0/22
  Latency: 16.7819ms
 netbird-a2sdv-1-1.netbird.selfhosted:
  NetBird IP: 100.71.188.65
  Public key: glJyPm+D1gLQYtABng2oGZCyQqLF5QRBrMripmcmYTg=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 172.26.128.1:51820/10.7.3.4:51820
  Relay server address:
  Last connection update: 41 seconds ago
  Last WireGuard handshake: 42 seconds ago
  Transfer status (received/sent) 1.2 KiB/1.7 KiB
  Quantum resistance: false
  Networks: 10.7.0.0/23, 10.7.3.1/32, 192.168.7.0/24, 198.51.100.3/32
  Latency: 6.706ms
 netbird-rpi5-1.netbird.selfhosted:
  NetBird IP: 100.71.246.55
  Public key: ZsqyMXTm0tRK3JCztt+lw51dnqo1BLZ6yKhZOMblj3E=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.4:22177/10.7.3.3:51820
  Relay server address:
  Last connection update: 41 seconds ago
  Last WireGuard handshake: 42 seconds ago
  Transfer status (received/sent) 92 B/212 B
  Quantum resistance: false
  Networks: -
  Latency: 5.5425ms
Events:
  [WARNING] DNS (0468300e-56bd-43a3-b296-54eeaf9e9547)
    Message: All upstream servers failed (probe failed)
    Time: 57 seconds ago
    Metadata: upstreams: 10.243.4.10:53
  [WARNING] DNS (18e46f77-b610-46a1-bd3c-783659dc476a)
    Message: All upstream servers failed (probe failed)
    Time: 57 seconds ago
    Metadata: upstreams: 10.243.4.10:53
  [WARNING] DNS (9d86a12d-0b40-4eb7-afd6-adba98e8ce4b)
    Message: All upstream servers failed (probe failed)
    Time: 57 seconds ago
    Metadata: upstreams: 10.231.230.2:53
  [INFO] SYSTEM (de01161f-67dc-4e9f-9d61-9ed71cc502f7)
    Message: Network map updated
    Time: 57 seconds ago
  [WARNING] DNS (c9310827-104d-4422-9f3e-1432725cf204)
    Message: All upstream servers failed (probe failed)
    Time: 42 seconds ago
    Metadata: upstreams: 10.7.0.1:53
  [WARNING] DNS (9f3a7006-22b7-4279-bede-39c0a4d1e3ac)
    Message: All upstream servers failed (probe failed)
    Time: 42 seconds ago
    Metadata: upstreams: 10.243.4.10:53
  [WARNING] DNS (76e4a1bd-29ae-4107-9ce2-0edf676ff332)
    Message: All upstream servers failed (probe failed)
    Time: 42 seconds ago
    Metadata: upstreams: 10.243.4.10:53
  [WARNING] DNS (913dc72d-a75f-4dad-898c-782d22891570)
    Message: All upstream servers failed (probe failed)
    Time: 42 seconds ago
    Metadata: upstreams: 10.231.230.2:53
  [WARNING] DNS (734434a8-ba11-467a-8381-b2315dbf87af)
    Message: All upstream servers failed (probe failed)
    Time: 42 seconds ago
    Metadata: upstreams: 10.7.0.1:53
  [INFO] SYSTEM (5f9de153-2967-4e75-aef5-9f9806a154ec)
    Message: Network map updated
    Time: 42 seconds ago
OS: windows/amd64
Daemon version: 0.45.1
CLI version: 0.45.1
Management: Connected to https://nb.anon-Rv9JS.domain:33073
Signal: Connected to http://nb.anon-Rv9JS.domain:10000
Relays:
  [stun:nb.anon-Rv9JS.domain:3478] is Available
  [turn:nb.anon-Rv9JS.domain:3478?transport=udp] is Available
Nameservers:
  [10.231.230.2:53] for [aws-uk-sbx-audiobroadcast.anon-iqBqM.domain] is Available
  [10.7.0.1:53] for [office.anon-Rv9JS.domain] is Available
  [10.7.0.1:53] for [k3s-devel.anon-Rv9JS.domain] is Available
  [10.243.4.10:53] for [argocd.svc.dev.gcp.anon-V5qtK.domain, jupyter.svc.dev.gcp.anon-V5qtK.domain] is Available
  [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: zbduo8406.netbird.selfhosted
NetBird IP: 100.71.137.187/16
Interface type: Userspace
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 4/6 Connected

Create and upload a debug bundle, and share the returned file key:

2c083968ec611b79db72ce4a8f4aae94746840c955f0ae82ae55b08e0a33a96b/a125425f-5f5e-4291-a028-dd3d820478bb

Screenshots

Image

Additional context

DNS search was working fine until the 19th of May, which is roughly when I upgraded routing peers to 0.40.0. This may be coincidental though as I have downgraded to 0.39.2 and DNS search functionality is not restored.

DNS search is still working for locally-connected clients, i.e. from my desktop I can resolve peer.example.com by doing nslookup peer, using the same name server. I believe that rules out an issue with the nameserver itself. Additionally, hosts in netbird.selfhosted do not resolve, which has nothing to do with my office nameserver.

This is affecting clients on Windows, Ubuntu and Android.

Have you tried these troubleshooting steps?

  • [x] Reviewed client troubleshooting (if applicable)
  • [x] Checked for newer NetBird versions
  • [x] Searched for similar issues on GitHub (including closed ones)
  • [x] Restarted the NetBird client
  • [ ] Disabled other VPN software
  • [x] Checked firewall settings

drewhemm avatar May 28 '25 21:05 drewhemm

This is interesting: On a Windows 11 machine with WSL 2, inside WSL's Ubuntu, /etc/resolv.conf is being managed by the Windows Netbird client. It is adding the search domain, and nslookup peer does return peer.example.com. I also published the nameserver to some Ubuntu peers and they are working too.

It's definitely not working inside Windows or Android though.

drewhemm avatar May 28 '25 22:05 drewhemm

On Windows, do users need to have administrative accounts for DNS search domains to work?

drewhemm avatar May 28 '25 22:05 drewhemm

I have found a workaround for Windows. It involves going in to the Group Policy and adding the search domain to Administrative Templates > Network > DNS Client > DNS suffix search list.

This definitely should not be necessary and does nothing to fix Android peers.

drewhemm avatar May 29 '25 21:05 drewhemm

For windows, can you test v0.46.0 please? For Android, can you test v0.1.0 (beta track), please?

lixmal avatar Jun 08 '25 13:06 lixmal