nebula icon indicating copy to clipboard operation
nebula copied to clipboard
verified publisher icon slackhq

🐛 BUG: DNS server returns inconsistent results on port 53

Open altano opened this issue 4 months ago • 2 comments

What version of nebula are you using? (nebula -version)

1.9.7

What operating system are you using?

Linux (NixOS)

Describe the Bug

Problem

My lighthouse DNS server:

  • works on port 5353 both remotely/locally
  • works on port 53 locally
  • responds on port 53 for remote requests but has no results

My setup

I have a lighthouse with DNS enabled at IP 10.2.2.2 and I have a node with name "builder".

The nebula interface is a trusted interface, so the system firewall (iptables) won't block any traffic on it.

Tests

With host = 10.2.2.2 and port = 5353: ✅ dig @10.2.2.2 -p 5353 +short builder A works from remote host ✅ dig @10.2.2.2 -p 5353 +short builder A works from local host

With host = 10.2.2.2 and port = 53: ❌ dig @10.2.2.2 -p 53 +short builder A DNS server responds, but no entries (no ANSWER SECTION) ✅ dig @10.2.2.2 -p 53 +short builder A works from local host

With host = 0.0.0.0 and port = 53: ❌ dig @10.2.2.2 -p 53 +short builder A DNS server responds, but no entries (no ANSWER SECTION) ✅ dig @10.2.2.2 -p 53 +short builder A works from local host

Nebula is the only thing listening on port 53:

# sudo lsof -nP -iUDP
COMMAND    PID           USER FD   TYPE DEVICE SIZE/OFF NODE NAME
nebula  150109 nebula-homelab 6u  IPv4 940404      0t0  UDP *:4242
nebula  150109 nebula-homelab 8u  IPv6 940413      0t0  UDP *:53

When dig returns results I see this in the lighthouse's logs (debug logging enabled):

Dec 08 21:06:38 nebula-homelab-cc nebula[150109]: time="2025-12-08T21:06:38-08:00" level=debug msg="Query for A builder."

And when dig returns no results I get no log entry.

Logs from affected hosts

Lighthouse logs:

Dec 08 21:13:09 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:09-08:00" level=debug msg="Tunnel status" certName=syncboy localIndex=134885180 remoteIndex=2728028678 tunnelCheck="map[method:passive state:alive]" vpnIp=10.2.2.14
Dec 08 21:13:10 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:10-08:00" level=debug msg="Tunnel status" certName=vpn-internet localIndex=4292451000 remoteIndex=4216527219 tunnelCheck="map[method:passive state:alive]" vpnIp=10.2.2.15
Dec 08 21:13:11 nebula-homelab-cc systemd[1]: Stopping Nebula VPN service for homelab...
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=info msg="Caught signal, shutting down" signal=terminated
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:2603499616 mapTotalSize:8 remoteIndexNumber:709223335 vpnIp:10.2.2.12]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.12 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<home-ip-redacted>:15697" vpnIp=10.2.2.12
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:3018463858 mapTotalSize:7 remoteIndexNumber:3237372178 vpnIp:10.2.2.10]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.10 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<public-ip-redacted>:49175" vpnIp=10.2.2.10
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:1615597557 mapTotalSize:6 remoteIndexNumber:2352893906 vpnIp:10.2.2.16]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.16 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<home-ip-redacted>:15698" vpnIp=10.2.2.16
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:1060824857 mapTotalSize:5 remoteIndexNumber:2981908504 vpnIp:10.2.2.19]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.19 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<home-ip-redacted>:15700" vpnIp=10.2.2.19
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:134885180 mapTotalSize:4 remoteIndexNumber:2728028678 vpnIp:10.2.2.14]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.14 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<public-ip-redacted>:28546" vpnIp=10.2.2.14
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:2915658197 mapTotalSize:3 remoteIndexNumber:2998918148 vpnIp:10.2.2.17]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.17 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.17
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:426040382 mapTotalSize:2 remoteIndexNumber:1161189072 vpnIp:10.2.2.13]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.13 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.13
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:3319116426 mapTotalSize:1 remoteIndexNumber:669647551 vpnIp:10.2.2.11]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.11 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.11
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap hostInfo deleted" hostMap="map[indexNumber:4292451000 mapTotalSize:0 remoteIndexNumber:4216527219 vpnIp:10.2.2.15]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="deleting 10.2.2.15 from lighthouse."
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=debug msg="Sending close tunnel message" udpAddr="<home-ip-redacted>:4242" vpnIp=10.2.2.15
Dec 08 21:13:11 nebula-homelab-cc nebula[150487]: time="2025-12-08T21:13:11-08:00" level=info msg=Goodbye
Dec 08 21:13:11 nebula-homelab-cc systemd[1]: [email protected]: Deactivated successfully.
Dec 08 21:13:11 nebula-homelab-cc systemd[1]: Stopped Nebula VPN service for homelab.
Dec 08 21:13:11 nebula-homelab-cc systemd[1]: [email protected]: Consumed 82ms CPU time, 4M memory peak, 15.8K incoming IP traffic, 9.9K outgoing IP traffic.
Dec 08 21:13:11 nebula-homelab-cc systemd[1]: Starting Nebula VPN service for homelab...
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Client nebula certificate" cert="<redacted>"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Trusted CA fingerprints" fingerprints="[<redacted>]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Firewall rule added" firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: localIp: proto:0 startPort:0]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Firewall rule added" firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: localIp: proto:1 startPort:0]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Firewall rule added" firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: localIp: proto:0 startPort:0]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Firewall started" firewallHashes="SHA:88a6373dd147f80c6760c0a080642a3fcec4f4b1d1d93ca2cfa3b7eb5fef8083,FNV:4144628004"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="listening on 0.0.0.0:4242"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Main HostMap created" network=10.2.2.2/24 preferredRanges="[]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="punchy enabled"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Loaded send_recv_error config" sendRecvError=always
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Starting dns server"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Nebula interface is active" boringcrypto=false build=1.9.7 interface=nebula.homelab network=10.2.2.2/24 udpAddr="0.0.0.0:4242"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Starting DNS responder" dnsListener="10.2.2.2:53"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="notified systemd the service is ready"
Dec 08 21:13:11 nebula-homelab-cc systemd[1]: Started Nebula VPN service for homelab.
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Error while validating outbound packet: packet is not ipv4, type: 6" packet="[<redacted>]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Generated index" index=377529692
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Handshake message received" certName=builder fingerprint=<redacted> handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2706135077 issuer=<redacted> remoteIndex=0 responderIndex=0 udpAddr="<public-ip-redacted>:49175" vpnIp=10.2.2.10
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap vpnIp added" hostMap="map[hostinfo:map[existing:true hostId:10.2.2.10 localIndexId:377529692] mapTotalSize:1 vpnIp:10.2.2.10]"

Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Handshake message sent" certName=builder fingerprint=<redacted> handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2706135077 issuer=<redacted> remoteIndex=0 responderIndex=377529692 udpAddr="<public-ip-redacted>:49175" vpnIp=10.2.2.10
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Generated index" index=1208105941
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Handshake message received" certName=vpn-homelab-house fingerprint=<redacted> handshake="map[stage:1 style:ix_psk0]" initiatorIndex=3464208539 issuer=<redacted> remoteIndex=0 responderIndex=0 udpAddr="<home-ip-redacted>:15698" vpnIp=10.2.2.16
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=debug msg="Hostmap vpnIp added" hostMap="map[hostinfo:map[existing:true hostId:10.2.2.16 localIndexId:1208105941] mapTotalSize:2 vpnIp:10.2.2.16]"
Dec 08 21:13:11 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:11-08:00" level=info msg="Handshake message sent" certName=vpn-homelab-house fingerprint=<redacted> handshake="map[stage:2 style:ix_psk0]" initiatorIndex=3464208539 issuer=<redacted> remoteIndex=0 responderIndex=1208105941 udpAddr="<home-ip-redacted>:15698" vpnIp=10.2.2.16
Dec 08 21:13:13 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:13-08:00" level=debug msg="Query for A builder."
Dec 08 21:13:13 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:13-08:00" level=debug msg="Generated index" index=279816397
Dec 08 21:13:13 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:13-08:00" level=info msg="Handshake message received" certName=lacon fingerprint=<redacted> handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2144086643 issuer=<redacted> remoteIndex=0 responderIndex=0 udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.13
Dec 08 21:13:13 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:13-08:00" level=debug msg="Hostmap vpnIp added" hostMap="map[hostinfo:map[existing:true hostId:10.2.2.13 localIndexId:279816397] mapTotalSize:3 vpnIp:10.2.2.13]"
Dec 08 21:13:13 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:13-08:00" level=info msg="Handshake message sent" certName=lacon fingerprint=<redacted> handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2144086643 issuer=<redacted> remoteIndex=0 responderIndex=279816397 udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.13

Dec 08 21:13:15 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:15-08:00" level=debug msg="Generated index" index=427504245
Dec 08 21:13:15 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:15-08:00" level=info msg="Handshake message received" certName=control fingerprint=<redacted> handshake="map[stage:1 style:ix_psk0]" initiatorIndex=4222215296 issuer=<redacted> remoteIndex=0 responderIndex=0 udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.11
Dec 08 21:13:15 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:15-08:00" level=debug msg="Hostmap vpnIp added" hostMap="map[hostinfo:map[existing:true hostId:10.2.2.11 localIndexId:427504245] mapTotalSize:4 vpnIp:10.2.2.11]"
Dec 08 21:13:15 nebula-homelab-cc nebula[150702]: time="2025-12-08T21:13:15-08:00" level=info msg="Handshake message sent" certName=control fingerprint=<redacted> handshake="map[stage:2 style:ix_psk0]" initiatorIndex=4222215296 issuer=<redacted> remoteIndex=0 responderIndex=427504245 udpAddr="<public-ip-redacted>:4242" vpnIp=10.2.2.11

My machine's logs (the "remote" node I refer to above):

time="2025-12-08T21:00:46-08:00" level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2970661025 localIndex=2970661025 remoteIndex=0 udpAddrs="[<lighthouse-ip-redacted>:4242]" vpnAddrs="[10.2.2.2]"
time="2025-12-08T21:00:46-08:00" level=info msg="Handshake message received" certName=nebula-homelab-cc certVersion=1 durationNs=24430834 fingerprint=<redacted> from="<lighthouse-ip-redacted>:4242" handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2970661025 issuer=<redacted> remoteIndex=2970661025 responderIndex=3107498205 sentCachedPackets=1 vpnAddrs="[10.2.2.2]"
time="2025-12-08T21:11:46-08:00" level=info msg="Close tunnel received, tearing down." certName=nebula-homelab-cc from="<lighthouse-ip-redacted>:4242" localIndex=2970661025 remoteIndex=3107498205 vpnAddrs="[10.2.2.2]"
time="2025-12-08T21:11:46-08:00" level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2981908504 localIndex=2981908504 remoteIndex=0 udpAddrs="[<lighthouse-ip-redacted>:4242]" vpnAddrs="[10.2.2.2]"
time="2025-12-08T21:11:46-08:00" level=info msg="Handshake message received" certName=nebula-homelab-cc certVersion=1 durationNs=101046333 fingerprint=<redacted> from="<lighthouse-ip-redacted>:4242" handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2981908504 issuer=<redacted> remoteIndex=2981908504 responderIndex=1060824857 sentCachedPackets=1 vpnAddrs="[10.2.2.2]"
time="2025-12-08T21:13:11-08:00" level=info msg="Close tunnel received, tearing down." certName=nebula-homelab-cc from="<lighthouse-ip-redacted>:4242" localIndex=2981908504 remoteIndex=1060824857 vpnAddrs="[10.2.2.2]"

Config files from affected hosts

Lighthouse config:

firewall:
  inbound:
  - host: any
    port: any
    proto: icmp
  - host: any
    port: any
    proto: any
  inbound_action: drop
  outbound:
  - host: any
    port: any
    proto: any
  outbound_action: drop
lighthouse:
  am_lighthouse: true
  dns:
    host: 10.2.2.2
    port: 53
  hosts: []
  serve_dns: true
listen:
  host: 0.0.0.0
  port: 4242
logging:
  level: debug
pki:
  ca: /nix/store/49wj7a9v8fxr4ihgysxnkppkm20c2vrk-ca.crt
  cert: /nix/store/yhb2vrf5y4iqirrabk2gsphpcbh201pd-nebula.crt
  disconnect_invalid: true
  key: /var/lib/nebula-homelab/nebula.key
punchy:
  punch: true
  respond: true
relay:
  am_relay: true
  relays: []
  use_relays: false
static_host_map: {}
tun:
  dev: nebula.homelab
  disabled: false

altano avatar Dec 09 '25 05:12 altano

I switched to using a tld in the cert name for all my hosts (I changed Name: macbook to Name: macbook.example.org) and this problem seems to have gone away.

I'm not 100% sure that's the only thing I changed, but I'm fairly confident.

altano avatar Dec 09 '25 07:12 altano 

With host = 10.2.2.2 and port = 5353:
✅ dig @10.2.2.2 -p 5353 +short builder A works from remote host
✅ dig @10.2.2.2 -p 5353 +short builder A works from local host

With host = 10.2.2.2 and port = 53:
❌ dig @10.2.2.2 -p 53 +short builder A DNS server responds, but no entries (no ANSWER SECTION)
✅ dig @10.2.2.2 -p 53 +short builder A works from local host

With host = 0.0.0.0 and port = 53:
❌ dig @10.2.2.2 -p 53 +short builder A DNS server responds, but no entries (no ANSWER SECTION)
✅ dig @10.2.2.2 -p 53 +short builder A works from local host

This behavior seems like you might have something wonky going in with traffic on port 53 getting redirected somehow. Is there any chance that's the case? I don't want to sound like I'm dismissing this or blaming your system, but Nebula doesn't care what port number you configure, it will always behave the same way.

Looking at your logs, it looks like Nebula is getting the DNS query. Is it possible something is trampling over your response?

For what it's worth, I haven't been able to reproduce this on 1.10, but the code paths aren't that different.

JackDoan avatar Dec 09 '25 16:12 JackDoan