netbird
netbird copied to clipboard
Published
20 hours ago •
netbirdio
netbirdio
Can only ping hosts on same network on self-hosted on Oracle
Describe the problem Can only ping other Netbird hosts on the same local network
To Reproduce
- Install Netbird self-hosted on Oracle Cloud Infrastructure (2 vCPU/12GB RAM/Ubuntu 22.04).
- Complete setup process
- Add peers on local and remote networks
- run
sudo iptables -I INPUT -p udp -m udp --dport 3478 -j ACCEPTas per https://docs.netbird.io/selfhosted/selfhosted-guide#oracle-cloud-infrastructure-oci - try to ping local hosts
- try to ping remote hosts
A clear and concise description of what you expected to happen. Pinging to work between networks
Are you using NetBird Cloud? No, using selfhosted
NetBird version 0.27.10
Additional context
All hosts are showing as green on the Netbird dashboard.
When running status -dA I note the clients I can't ping are showing as offline in that output despite being online on the dashboard.
NetBird status -dA output
Peers detail:
pixel8.netbird.selfhosted:
NetBird IP: 100.72.18.3
Public key: spdafpxNQ9EeM3tPZq3J59T50P5C3/qnxFD72ZaLbQg=
Status: Disconnected
-- detail --
Connection type: P2P
Direct: false
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
Last connection update: 19 hours, 11 minutes ago
Last WireGuard handshake: 2 minutes, 4 seconds ago
Transfer status (received/sent) 5.9 KiB/1.8 KiB
Quantum resistance: false
Routes: -
Latency: 0s
chillblast-edi.netbird.selfhosted:
NetBird IP: 100.72.63.91
Public key: VZfY5WgbKibJNCU/WzMqoX8hBn9iyC3YxoTqiizSmwM=
Status: Disconnected
-- detail --
Connection type: P2P
Direct: false
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
Last connection update: 19 hours, 28 minutes ago
Last WireGuard handshake: 2 minutes, 4 seconds ago
Transfer status (received/sent) 5.9 KiB/1.8 KiB
Quantum resistance: false
Routes: -
Latency: 39.862337ms
pi4.netbird.selfhosted:
NetBird IP: 100.72.142.196
Public key: xoSpZtj5S4NADpy/Ln1L9X7T2KwG3QvEX6h5/04onwQ=
Status: Connecting
-- detail --
Connection type: P2P
Direct: false
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
Last connection update: 8 seconds ago
Last WireGuard handshake: 2 minutes, 4 seconds ago
Transfer status (received/sent) 5.9 KiB/1.8 KiB
Quantum resistance: false
Routes: -
Latency: 0s
chillblast-dnd.netbird.selfhosted:
NetBird IP: 100.72.158.226
Public key: BgIDgECerz62sgaC4U4dSLW8MvEXGBr0PGu62TH7bwA=
Status: Connected
-- detail --
Connection type: P2P
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
Last connection update: 37 minutes, 30 seconds ago
Last WireGuard handshake: 2 minutes, 4 seconds ago
Transfer status (received/sent) 5.9 KiB/1.8 KiB
Quantum resistance: false
Routes: -
Latency: 1.614116ms
OS: linux/arm (ARMv)
Daemon version: 0.27.10
CLI version: 0.27.10
Management: Connected to <https://network.anon-aHrXN.domain:443>
Signal: Connected to <https://network.anon-aHrXN.domain:443>
Relays:
[stun:network.anon-aHrXN.domain:3478] is Unavailable, reason: stun request: context deadline exceeded
[turn:network.anon-aHrXN.domain:3478?transport=udp] is Unavailable, reason: allocate: all retransmissions failed for xIkecmxLzfjXr1y/
Nameservers:
FQDN: pi400.netbird.selfhosted
NetBird IP: 100.72.103.36/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 1/4 Connected```
[Slack Message](https://netbirdio.slack.com/archives/C05T5K65X7U/p1718045885486609)
@EDIflyer it seems like your relay service is not reachable. Can you run the test from this page? https://docs.netbird.io/selfhosted/troubleshooting
Hi @mlsmaycon sorry for the slow reply, been tied up with work. Have tried it out and I get the following (server name redacted)...
Note: errors from onicecandidateerror above are not necessarily fatal. For example an IPv6 DNS lookup may fail but relay candidates can still be gathered via IPv4.
The server stun:netbird.<MYDOMAIN.COM>:3478 returned an error with code=701:
STUN host lookup received error.
The server turn:netbird.<MYDOMAIN.COM>:3478?transport=udp returned an error with code=701:
TURN host lookup received error.
The server stun:netbird.<MYDOMAIN.COM>:3478 returned an error with code=701:
STUN binding request timed out.
The server turn:netbird.<MYDOMAIN.COM>:3478?transport=udp returned an error with code=701:
TURN allocate request timed out.
As far as I can tell from sudo ss -atpu I have connections coming in port 3478 OK?
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.53%lo:domain 0.0.0.0:* users:(("systemd-resolve",pid=657,fd=13))
udp UNCONN 0 0 10.0.0.67%enp0s6:bootpc 0.0.0.0:* users:(("systemd-network",pid=655,fd=15))
udp UNCONN 0 0 0.0.0.0:sunrpc 0.0.0.0:* users:(("rpcbind",pid=594,fd=5),("systemd",pid=1,fd=141))
udp UNCONN 0 0 172.18.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=32))
udp UNCONN 0 0 172.18.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=31))
udp UNCONN 0 0 172.17.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=30))
udp UNCONN 0 0 172.17.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=29))
udp UNCONN 0 0 10.0.0.67:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=28))
udp UNCONN 0 0 10.0.0.67:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=27))
udp UNCONN 0 0 127.0.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=26))
udp UNCONN 0 0 127.0.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=25))
udp UNCONN 0 0 [::]:sunrpc [::]:* users:(("rpcbind",pid=594,fd=7),("systemd",pid=1,fd=143))
udp UNCONN 0 0 [::1]:3478 [::]:* users:(("turnserver",pid=4175,fd=33))
udp UNCONN 0 0 [::1]:3478 [::]:* users:(("turnserver",pid=4175,fd=34))
tcp LISTEN 0 1024 172.17.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=22))
tcp LISTEN 0 1024 172.17.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=13))
tcp LISTEN 0 1024 127.0.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=20))
tcp LISTEN 0 1024 127.0.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=11))
tcp LISTEN 0 128 0.0.0.0:ssh 0.0.0.0:* users:(("sshd",pid=780,fd=3))
tcp LISTEN 0 4096 0.0.0.0:http 0.0.0.0:* users:(("docker-proxy",pid=3423,fd=4))
tcp LISTEN 0 4096 0.0.0.0:sunrpc 0.0.0.0:* users:(("rpcbind",pid=594,fd=4),("systemd",pid=1,fd=140))
tcp LISTEN 0 4096 0.0.0.0:https 0.0.0.0:* users:(("docker-proxy",pid=3404,fd=4))
tcp LISTEN 0 1024 10.0.0.67:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=21))
tcp LISTEN 0 1024 10.0.0.67:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=12))
tcp LISTEN 0 1024 172.18.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=23))
tcp LISTEN 0 1024 172.18.0.1:3478 0.0.0.0:* users:(("turnserver",pid=4175,fd=14))
tcp LISTEN 0 4096 0.0.0.0:http-alt 0.0.0.0:* users:(("docker-proxy",pid=3385,fd=4))
tcp LISTEN 0 4096 127.0.0.53%lo:domain 0.0.0.0:* users:(("systemd-resolve",pid=657,fd=14))
tcp ESTAB 0 0 10.0.0.67:ssh (redacted):54251 users:(("sshd",pid=826757,fd=4),("sshd",pid=826669,fd=4))
tcp LISTEN 0 128 [::]:ssh [::]:* users:(("sshd",pid=780,fd=4))
tcp LISTEN 0 4096 [::]:http [::]:* users:(("docker-proxy",pid=3430,fd=4))
tcp LISTEN 0 4096 [::]:sunrpc [::]:* users:(("rpcbind",pid=594,fd=6),("systemd",pid=1,fd=142))
tcp LISTEN 0 4096 [::]:https [::]:* users:(("docker-proxy",pid=3410,fd=4))
tcp LISTEN 0 1024 [::1]:3478 [::]:* users:(("turnserver",pid=4175,fd=15))
tcp LISTEN 0 1024 [::1]:3478 [::]:* users:(("turnserver",pid=4175,fd=24))
tcp LISTEN 0 4096 [::]:http-alt [::]:* users:(("docker-proxy",pid=3392,fd=4))
and from sudo iptables --list...
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:3478
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT udp -- anywhere anywhere udp spt:ntp
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
InstanceServices all -- anywhere link-local/16
Chain DOCKER (2 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.18.0.3 tcp dpt:http-alt
ACCEPT tcp -- anywhere 172.18.0.3 tcp dpt:https
ACCEPT tcp -- anywhere 172.18.0.3 tcp dpt:http
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain InstanceServices (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 169.254.0.2 owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.2.0/24 owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.4.0/24 owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.5.0/24 owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.0.2 tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT udp -- anywhere 169.254.169.254 udp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.169.254 tcp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.0.3 owner UID match root tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.0.4 tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT tcp -- anywhere 169.254.169.254 tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT udp -- anywhere 169.254.169.254 udp dpt:bootps /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT udp -- anywhere 169.254.169.254 udp dpt:tftp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT udp -- anywhere 169.254.169.254 udp dpt:ntp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
REJECT tcp -- anywhere link-local/16 tcp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with tcp-reset
REJECT udp -- anywhere link-local/16 udp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with icmp-port-unreachable
Just to update - to be doubly-certain all the required ports were open I've run the following commands:
sudo iptables -A INPUT -p tcp -m multiport --dports 80,443,10000,33073 -j ACCEPT
sudo iptables -A INPUT -p udp -m multiport --dports 3478,49152:65535 -j ACCEPT
However I'm still having the same issue that I can only ping devices on the same local network.
Here's the coturn docker logs too.
I do note that the domain name is set to netbird.relay.selfhosted (which isn't the FQDN but I'm not sure if it's some sort of default?) and realm of wiretrustee.com (same)?
@mlsmaycon still nothing the TURN tester either
Hello @EDIflyer,
We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.
Could you please confirm if the issue is still there?
We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.
Thanks for your contribution to improving the project!
Hi @nazarewk sorry I stopped using Netbird as I couldn't get it to work for me so feel free to close.
@nazarewk I am not able to get STUN to work either. No matter what settings are applied I still end up with the same error. This is on the latest 0.58.9 server and client
