netbird
netbird copied to clipboard
netbirdio
Docker container or K8s pod do not send request to host, but correct resolve host to ip.
Describe the problem
Netbird in docker container or in kubernetes pod for self hosted netbird. Dns resolve correct ip (nslookup works correct) but curl, ping, wget return error: bad address For normal ubuntu instance without docker or kubernetes evereythings works correct
How I run docker
docker run --rm -d
--cap-add=NET_ADMIN
-e NB_SETUP_KEY=<REDACTED>
-v netbird-client:/etc/netbird
-e NB_MANAGEMENT_URL=<REDACTED>
netbirdio/netbird:latest
How I run k8s pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: netbird
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: netbird
template:
metadata:
labels:
app: netbird
spec:
containers:
- name: netbird
image: netbirdio/netbird:latest
env:
- name: NB_MANAGEMENT_URL
value: "<REDACTED>"
- name: NB_SETUP_KEY
value: "<REDACTED>" # replace with your setup key
securityContext:
capabilities:
add:
- NET_ADMIN
Example commands executed in docker container or in k8s pods. Nslookup return correct ip 100.87.97.114
nslookup netbird-client-1-2.netbird.selfhosted
Server: 100.87.150.177
Address: 100.87.150.177:53
** server can't find netbird-client-1-2.netbird.selfhosted: NXDOMAIN
Non-authoritative answer:
Name: netbird-client-1-2.netbird.selfhosted
Address: 100.87.97.114
Ping with ip works correct
/ # ping 100.87.97.114
PING 100.87.97.114 (100.87.97.114): 56 data bytes
64 bytes from 100.87.97.114: seq=0 ttl=64 time=1.503 ms
64 bytes from 100.87.97.114: seq=1 ttl=64 time=1.032 ms
Wget with ip works correct
wget 100.87.97.114:5555
Connecting to 100.87.97.114:5555 (100.87.97.114:5555)
saving to 'index.html'
index.html 100% |******************************************************| 5 0:00:00 ETA
'index.html' saved
Error here
But when I execute client http request or ping for host I received dns resolve error:
ping netbird-client-1-2.netbird.selfhosted
ping: bad address 'netbird-client-1-2.netbird.selfhosted'
wget netbird-client-1-2.netbird.selfhosted
wget: bad address 'netbird-client-1-2.netbird.selfhosted'
wget netbird-client-1-2:5555
wget: bad address 'netbird-client-1-2:5555'
Expected behavior
Ping, curl or wget should send request
Are you using NetBird Cloud?
Netbird self hosted
NetBird version
0.27.10
NetBird status -d output:
I run netbird in container, I cant execute status output
Hello @gawsoftpl, can you try testing again running client with these additional capabilities: --cap-add=SYS_ADMIN and --cap-add=SYS_RESOURCE ?
Hello @gawsoftpl, can you try testing again running client with these additional capabilities:
--cap-add=SYS_ADMINand--cap-add=SYS_RESOURCE?
This not help in docker and k8s same issue.
I past resolv conf
cat /etc/resolv.conf
# Generated by NetBird
# If needed you can restore the original file by copying back /etc/resolv.conf.original.netbird
options timeout:4 attempts:1
search netbird.selfhosted
nameserver 100.87.0.130
nameserver 100.87.231.247
nameserver 185.12.64.1
nameserver 185.12.64.2
cat /etc/nsswitch.conf
# musl itself does not support NSS, however some third-party DNS
# implementations use the nsswitch.conf file to determine what
# policy to follow.
# Editing this file is not recommended.
hosts: files dns
cat /etc/resolv.conf.original.netbird
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.
nameserver 100.87.231.247
nameserver 185.12.64.1
nameserver 185.12.64.2
search netbird.selfhosted
# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: []
And ifconfig from docker container
eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02
inet addr:172.17.0.2 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:948 errors:0 dropped:0 overruns:0 frame:0
TX packets:1107 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:135010 (131.8 KiB) TX bytes:137313 (134.0 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:39 errors:0 dropped:0 overruns:0 frame:0
TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3898 (3.8 KiB) TX bytes:3898 (3.8 KiB)
wt0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:100.87.0.130 P-t-P:100.87.0.130 Mask:255.255.0.0
UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:18 errors:5 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1364 (1.3 KiB) TX bytes:1928 (1.8 KiB)
@gawsoftpl you can enable the debug logs with -e NB_LOG_LEVEL=debug and share the client logs
I have the same issue in Kubernetes.
The DNS resolution isn't consistent. There are tools that work, and there are tools that don't.
For example, working for me are:
- dig
- curl
Not working are:
- wget
- curl
nslookup is somewhere in the middle, it does actually return the right IP, but it also throws an NXDOMAIN.
nslookup -debug myhost.netbird.selfhosted
Server: 10.43.0.10
Address: 10.43.0.10#53
------------
QUESTIONS:
myhost.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1736502945
refresh = 7200
retry = 1800
expire = 86400
minimum = 5
ttl = 5
ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN
Server: 10.43.0.10
Address: 10.43.0.10#53
------------
QUESTIONS:
myhost.netbird.selfhosted.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1736502945
refresh = 7200
retry = 1800
expire = 86400
minimum = 5
ttl = 5
ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted.svc.cluster.local: NXDOMAIN
Server: 10.43.0.10
Address: 10.43.0.10#53
------------
QUESTIONS:
myhost.netbird.selfhosted.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1736502945
refresh = 7200
retry = 1800
expire = 86400
minimum = 5
ttl = 5
ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted.cluster.local: NXDOMAIN
Server: 10.43.0.10
Address: 10.43.0.10#53
------------
QUESTIONS:
myhost.netbird.selfhosted, type = A, class = IN
ANSWERS:
-> myhost.netbird.selfhosted
internet address = 100.69.86.108
ttl = 300
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name: myhost.netbird.selfhosted
Address: 100.69.86.108
------------
QUESTIONS:
myhost.netbird.selfhosted, type = AAAA, class = IN
ANSWERS:
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted: NXDOMAIN
An here dig:
dig myhost.netbird.selfhosted
; <<>> DiG 9.18.25 <<>> myhost.netbird.selfhosted
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13706
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 710e26c6b25cfdd2 (echoed)
;; QUESTION SECTION:
;myhost.netbird.selfhosted. IN A
;; ANSWER SECTION:
myhost.netbird.selfhosted. 300 IN A 100.69.86.108
;; Query time: 0 msec
;; SERVER: 10.43.0.10#53(10.43.0.10) (UDP)
;; WHEN: Fri Jan 10 10:14:02 UTC 2025
;; MSG SIZE rcvd: 107
So there seems to be something wonky if the DNS lookups happen within a container for some reason.
Hello @gawsoftpl,
We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.
Could you please confirm if the issue is still there?
We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.
Thanks for your contribution to improving the project!
@nazarewk
Can config, still the same behaviour as in my comment above. Just tested with Netbird 0.43.0.