netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Published 20 hours ago • verified publisher icon netbirdio

DNS resolution with Nameserver behind bastion host stopped working after Client update 0.46 -> 0.48 (Ubuntu)

Open Jachimoo opened this issue 4 months ago • 0 comments

Describe the problem

After updating client on ubuntu 22.04, the DNS resolution stopped working. The DNS nameservers for domain are configured and accessible but client

To Reproduce

Steps to reproduce the behavior:

  1. Have a selfhosted netbird instance working as bastion host, with DNS servers that extends our public domain to internal servers
  2. Configure Nameservers for client group to be set as internal servers for our domain (We use FreeIPA)
  3. Client v0.46 works fine
  4. Client v0.48 in unable to use internal DNS (Logs says that there is a timeout, but if we check manually the DNS server is accessible and resolves names)

Expected behavior

The nameservers are used without any problems

Are you using NetBird Cloud?

No I'm using selfhosted

NetBird version Server:

  • Control plane: latest as of 20.06.2025
  • Client: 0.45.1

Client: Works: 0.46.0 Broken: 0.48.0

Is any other VPN software installed?

Nope

Debug output (domain obfustated) client 0.46:

Peers detail:
  <some-client>anon-ZmCZH.domain:
  NetBird IP: 100.125.60.209/32
  Public key: VZMrvHq28JZD2xB7G+EuJ2QpME+vQVgOFfpjBrQRumI=
  Status: Idle
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 minute, 34 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 <some-client>.anon-ZmCZH.domain:
  NetBird IP: 100.125.36.234/32
  Public key: 0NguuRsxno4LMfvlkljqOhkFdXLlEPAigLiKOP3C308=
  Status: Idle
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 minute, 34 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 <routing_peer/control_plane>.anon-ZmCZH.domain:
  NetBird IP: 100.125.37.253
  Public key: cl2RJM9i3oqDb0422dzOa1SOU+9OXXV5zHKZrGR9u1c=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rel://netbird.anon-ZmCZH.domain:33080
  Last connection update: 1 minute, 33 seconds ago
  Last WireGuard handshake: 1 minute, 28 seconds ago
  Transfer status (received/sent) 9.9 KiB/8.5 KiB
  Quantum resistance: false
  Networks: 10.10.0.0/16, 10.100.0.0/16, 198.51.100.0/32
  Latency: 0s

 <some-cient>anon-ZmCZH.domain:
  NetBird IP: 100.125.51.179
  Public key: Dygbc4m2GoCyJClp6mCEs7tMg3e4gSnDgE8B+HKA9AY=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.122.1:51820/198.51.100.1:16135
  Relay server address: rel://netbird.anon-ZmCZH.domain:33080
  Last connection update: 1 minute, 33 seconds ago
  Last WireGuard handshake: 1 minute, 28 seconds ago
  Transfer status (received/sent) 424 B/364 B
  Quantum resistance: false
  Networks: -
  Latency: 14.349299ms

 <routing_peer>.anon-ZmCZH.domain:
  NetBird IP: 100.125.111.182
  Public key: kOOSzYtKp3kNB7vbgM6ge06DdIHPCds3a3YepEJFR2Q=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rel://netbird.anon-ZmCZH.domain:33080
  Last connection update: 1 minute, 4 seconds ago
  Last WireGuard handshake: 41 seconds ago
  Transfer status (received/sent) 572 B/488 B
  Quantum resistance: false
  Networks: 172.16.0.0/16, anon-LNzcq.domain
  Latency: 29.403567ms

Events:
  [WARNING] DNS (6560e27f-0c7b-441a-bd86-aba3c2b48691)
    Message: All upstream servers failed (probe failed)
    Time: 1 minute, 34 seconds ago
    Metadata: upstreams: 10.10.20.253:53, 10.10.30.253:53
  [WARNING] DNS (1f12b277-6c07-4982-9947-40584e5d2080)
    Message: All upstream servers failed (probe failed)
    Time: 1 minute, 34 seconds ago
    Metadata: upstreams: 10.10.20.253:53, 10.10.30.253:53
  [WARNING] DNS (e8058d77-f32e-4d00-82bf-09f142e2b958)
    Message: All upstream servers failed (probe failed)
    Time: 1 minute, 34 seconds ago
    Metadata: upstreams: 10.10.20.253:53, 10.10.30.253:53
  [INFO] SYSTEM (11ddcb9b-4c1f-41f7-8a8e-bb1f35a8a926)
    Message: Network map updated
    Time: 1 minute, 34 seconds ago
OS: linux/amd64
Daemon version: 0.46.0
CLI version: 0.46.0
Management: Connected to https://netbird.anon-ZmCZH.domain:443
Signal: Connected to https://netbird.anon-ZmCZH.domain:443
Relays: 
  [stun:netbird.anon-ZmCZH.domain:3478] is Available
  [turn:netbird.anon-ZmCZH.domain:3478?transport=udp] is Available
  [rel://netbird.anon-ZmCZH.domain:33080] is Available
Nameservers: 
  [10.10.20.253:53, 10.10.30.253:53] for [anon-ZmCZH.domain, anon-LNzcq.domain, anon-Xm8Aa.domain] is Available
FQDN: <client1>.anon-ZmCZH.domain
NetBird IP: 100.125.212.195/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 3/5 Connected


dig  <some-app>.anon-ZmCZH.domain                                                                                                                                                                

; <<>> DiG 9.18.30-0ubuntu0.22.04.2-Ubuntu <<>> <some-app>.anon-ZmCZH.domain
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36373
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;<some-app>.anon-ZmCZH.domain.		IN	A

;; ANSWER SECTION:
<some-app>.anon-ZmCZH.domain.	1200	IN	A	10.100.x.x //obfustated cuz dont need

;; AUTHORITY SECTION:
anon-ZmCZH.domain		86400	IN	NS	<FreeIpa server>
anon-ZmCZH.domain		86400	IN	NS	<FreeIpa server 2>

;; ADDITIONAL SECTION:
<FreeIpa server>	1200	IN	A	10.10.20.253
<FreeIpa server 2>	1200	IN	A	10.10.30.253

;; Query time: 32 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Jun 20 12:24:18 CEST 2025
;; MSG SIZE  rcvd: 133

client 0.48.:

Peers detail:
 <some-client>.anon-w4kgQ.domain:
  NetBird IP: 100.125.60.209/32
  Public key: VZMrvHq28JZD2xB7G+EuJ2QpME+vQVgOFfpjBrQRumI=
  Status: Idle
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 24 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 <some-client>.anon-w4kgQ.domain:
  NetBird IP: 100.125.36.234/32
  Public key: 0NguuRsxno4LMfvlkljqOhkFdXLlEPAigLiKOP3C308=
  Status: Idle
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 24 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

  <routing_peer/control_plane>.anon-w4kgQ.domain:
  NetBird IP: 100.125.37.253
  Public key: cl2RJM9i3oqDb0422dzOa1SOU+9OXXV5zHKZrGR9u1c=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rel://netbird.anon-w4kgQ.domain:33080
  Last connection update: 24 seconds ago
  Last WireGuard handshake: 18 seconds ago
  Transfer status (received/sent) 7.2 KiB/5.9 KiB
  Quantum resistance: false
  Networks: 10.10.0.0/16, 10.100.0.0/16, 198.51.100.0/32
  Latency: 0s

 <some-client>.anon-w4kgQ.domain:
  NetBird IP: 100.125.51.179
  Public key: Dygbc4m2GoCyJClp6mCEs7tMg3e4gSnDgE8B+HKA9AY=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.122.1:51820/198.51.100.1:4055
  Relay server address: rel://netbird.anon-w4kgQ.domain:33080
  Last connection update: 22 seconds ago
  Last WireGuard handshake: 18 seconds ago
  Transfer status (received/sent) 328 B/364 B
  Quantum resistance: false
  Networks: -
  Latency: 13.074887ms

 <routing_peer>.anon-w4kgQ.domain:
  NetBird IP: 100.125.111.182
  Public key: kOOSzYtKp3kNB7vbgM6ge06DdIHPCds3a3YepEJFR2Q=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.2:51820/198.51.100.0:51820
  Relay server address: rel://netbird.anon-w4kgQ.domain:33080
  Last connection update: 22 seconds ago
  Last WireGuard handshake: 23 seconds ago
  Transfer status (received/sent) 180 B/272 B
  Quantum resistance: false
  Networks: 172.16.0.0/16, anon-xfIRx.domain
  Latency: 33.40652ms

Events:
  [WARNING] DNS (1e0ef040-2d12-4c99-9b29-5a9d0d5ceef8)
    Message: All upstream servers failed (probe failed)
    Time: 24 seconds ago
    Metadata: upstreams: 10.10.20.253:53, 10.10.30.253:53
  [WARNING] DNS (783bb17c-2959-44a6-91e2-4d8448fa278f)
    Message: All upstream servers failed (probe failed)
    Time: 24 seconds ago
    Metadata: upstreams: 10.10.20.253:53, 10.10.30.253:53
  [WARNING] DNS (5e08b96c-38ba-483f-b945-867d74721478)
    Message: All upstream servers failed (probe failed)
    Time: 24 seconds ago
    Metadata: upstreams: 10.10.20.253:53, 10.10.30.253:53
  [INFO] SYSTEM (2b6f86a9-90e7-42d9-a67a-572a542bb26e)
    Message: Network map updated
    Time: 24 seconds ago
OS: linux/amd64
Daemon version: 0.48.0
CLI version: 0.48.0
Management: Connected to https://netbird.anon-w4kgQ.domain:443
Signal: Connected to https://netbird.anon-w4kgQ.domain:443
Relays: 
  [stun:netbird.anon-w4kgQ.domain:3478] is Unavailable, reason: dial: failed to listen: d.Dialer.DialContext: dial udp: lookup netbird.anon-w4kgQ.domain on 127.0.0.53:53: no such host
  [turn:netbird.anon-w4kgQ.domain:3478?transport=udp] is Unavailable, reason: create client: lookup netbird.anon-w4kgQ.domain on 127.0.0.53:53: no such host
  [rel://netbird.anon-w4kgQ.domain:33080] is Available
Nameservers: 
  [10.10.20.253:53, 10.10.30.253:53] for [anon-w4kgQ.domain, anon-xfIRx.domain, anon-OQqQX.domain] is Available
FQDN: gabriel-xps-13-7390.anon-w4kgQ.domain
NetBird IP: 100.125.212.195/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 3/5 Connected

dig  <some-app>.anon-ZmCZH.domain                                                                                                                                                                  

; <<>> DiG 9.18.30-0ubuntu0.22.04.2-Ubuntu <<>>  <some-app>.anon-ZmCZH.domain
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 47495
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
; <some-app>.anon-ZmCZH.domain		IN	A

;; Query time: 1 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Jun 20 13:07:23 CEST 2025
;; MSG SIZE  rcvd: 45

dig @10.10.20.253 <some-app>.anon-ZmCZH.domain                                                                                                                                                     

; <<>> DiG 9.18.30-0ubuntu0.22.04.2-Ubuntu <<>> @10.10.20.253 <some-app>.anon-ZmCZH.domain
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24850
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 1a4fe68f062fd27233556262685540fbea66c3affa429eeb (good)
;; QUESTION SECTION:
;<some-app>.anon-ZmCZH.domain.	IN	A

;; ANSWER SECTION:
<some-app>.anon-ZmCZH.domain	1200	IN	A	10.100.x.x //obfustated cuz dont need

;; AUTHORITY SECTION:
anon-ZmCZH.domain		86400	IN	NS	<FreeIpa server>
anon-ZmCZH.domain		86400	IN	NS	<FreeIpa server 2>

;; ADDITIONAL SECTION:
<FreeIpa server>	1200	IN	A	10.10.20.253
<FreeIpa server 2>	1200	IN	A	10.10.30.253

;; Query time: 33 msec
;; SERVER: 10.10.20.253#53(10.10.20.253) (UDP)
;; WHEN: Fri Jun 20 13:07:39 CEST 2025
;; MSG SIZE  rcvd: 161

The resolution is broken on client level, just for sure I've added a netbird.anon-ZmCZH.domain. record on my DNS server (thats just a NS record for Cloudflare domain):

dig netbird.anon-ZmCZH.domain.                                                                                                                                                               

; <<>> DiG 9.18.30-0ubuntu0.22.04.2-Ubuntu <<>> netbird.anon-ZmCZH.domain.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40036
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;netbird.anon-ZmCZH.domain..		IN	A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Jun 20 13:21:37 CEST 2025
;; MSG SIZE  rcvd: 47


dig @10.10.20.253 netbird.anon-ZmCZH.domain.                                                                                                                                                  

; <<>> DiG 9.18.30-0ubuntu0.22.04.2-Ubuntu <<>> @10.10.20.253 netbird.anon-ZmCZH.domain.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65409
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: bcbecf53da95a288f5add07568554492eba8f773f270ccaa (good)
;; QUESTION SECTION:
;netbird.anon-ZmCZH.domain.		IN	A

;; ANSWER SECTION:
netbird.anon-ZmCZH.domain.	275	IN	A	<NETBIRD PUBLIC IP>

;; AUTHORITY SECTION:
netbird.anon-ZmCZH.domain.	86400	IN	NS	rose.ns.cloudflare.com.
netbird.anon-ZmCZH.domain.	86400	IN	NS	sid.ns.cloudflare.com.

;; Query time: 34 msec
;; SERVER: 10.10.20.253#53(10.10.20.253) (UDP)
;; WHEN: Fri Jun 20 13:22:58 CEST 2025
;; MSG SIZE  rcvd: 142

Debug Files

netbird.debug-0.46.zip netbird.debug-0.48.zip

Additional context

Have you tried these troubleshooting steps?

  • [x] Reviewed client troubleshooting (if applicable)
  • [x] Checked for newer NetBird versions
  • [x] Searched for similar issues on GitHub (including closed ones)
  • [x] Restarted the NetBird client
  • [x] Disabled other VPN software
  • [x] Checked firewall settings

Jachimoo avatar Jun 20 '25 11:06 Jachimoo