netbird
netbird copied to clipboard
netbirdio
Unable to connect between peers
NETBIRD_DASHBOARD_TAG="v1.17.13" NETBIRD_SIGNAL_TAG="0.25.2" NETBIRD_MANAGEMENT_TAG="0.25.2" COTURN_TAG="latest"
Unable to connect between peers
[root@iZbp1imzcyvws0523mzrg4Z ~]# ping 10.255.249.205
PING 10.255.249.205 (10.255.249.205) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
acl is default !!
network routes is effic !!
root@iZbp11fpsa4uaxkx6jwliuZ:~# docker logs -f 666e48d2aa05
0: (1): INFO: System cpu num is 2
0: (1): INFO: log file opened: /var/tmp/turn_1_2023-12-27.log
0: (1): INFO: System enable num is 1
0: (1): INFO: Coturn Version Coturn-4.6.2 'Gorst'
0: (1): INFO: Coturn Version Coturn-4.6.2 'Gorst'
0: (1): INFO: Max number of open files/sockets allowed for this process: 1048576
0: (1): INFO: Due to the open files/sockets limitation, max supported number of TURN Sessions possible is: 524000 (approximately)
0: (1): INFO:
==== Show him the instruments, Practical Frost: ====
0: (1): INFO: OpenSSL compile-time version: OpenSSL 3.0.11 19 Sep 2023 (0x300000b0)
0: (1): INFO: TLS 1.3 supported
0: (1): INFO: DTLS 1.2 supported
0: (1): INFO: TURN/STUN ALPN supported
0: (1): INFO: Third-party authorization (oAuth) supported
0: (1): INFO: GCM (AEAD) supported
0: (1): INFO: SQLite supported, default database location is /var/lib/coturn/turndb
0: (1): INFO: Redis supported
0: (1): INFO: PostgreSQL supported
0: (1): INFO: MySQL supported
0: (1): INFO: MongoDB supported
0: (1): INFO: Default Net Engine version: 3 (UDP thread per CPU core)
0: (1): INFO: Domain name: netbird.xxxxx.cn
0: (1): INFO: Default realm: wiretrustee.com
0: (1): WARNING: cannot find certificate file: /etc/coturn/certs/cert.pem (1)
0: (1): WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
0: (1): WARNING: cannot find private key file: /etc/coturn/private/privkey.pem (1)
0: (1): WARNING: cannot start TLS and DTLS listeners because private key file is not set properly
0: (1): INFO: Certificate file found: /etc/coturn/certs/cert.pem
0: (1): INFO: Private key file found: /etc/coturn/private/privkey.pem
0: (1): WARNING: NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
0: (1): INFO: ===========Discovering listener addresses: =========
0: (1): INFO: Listener address to use: 127.0.0.1
0: (1): INFO: Listener address to use: 192.168.14.7
0: (1): INFO: Listener address to use: 172.17.0.1
0: (1): INFO: Listener address to use: 172.24.0.1
0: (1): INFO: Listener address to use: ::1
0: (1): INFO: =====================================================
0: (1): INFO: Total: 3 'real' addresses discovered
0: (1): INFO: =====================================================
0: (1): WARNING: NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
0: (1): INFO: ===========Discovering relay addresses: =============
0: (1): INFO: Relay address to use: 192.168.14.7
0: (1): INFO: Relay address to use: 172.17.0.1
0: (1): INFO: Relay address to use: 172.24.0.1
0: (1): INFO: Relay address to use: ::1
0: (1): INFO: =====================================================
0: (1): INFO: Total: 4 relay addresses discovered
0: (1): INFO: =====================================================
0: (1): INFO: pid file created: /var/tmp/turnserver.pid
0: (1): INFO: IO method: epoll (with changelist)
0: (1): INFO: Wait for relay ports initialization...
0: (1): INFO: relay 192.168.14.7 initialization...
0: (1): INFO: relay 192.168.14.7 initialization done
0: (1): INFO: relay 172.17.0.1 initialization...
0: (1): INFO: relay 172.17.0.1 initialization done
0: (1): INFO: relay 172.24.0.1 initialization...
0: (1): INFO: relay 172.24.0.1 initialization done
0: (1): INFO: relay ::1 initialization...
0: (1): INFO: relay ::1 initialization done
0: (1): INFO: Relay ports initialization done
0: (1): INFO: Total General servers: 2
9: (9): DEBUG: turn server id=1 created
9: (8): DEBUG: turn server id=0 created
9: (1): INFO: Total auth threads: 3
9: (1): INFO: prometheus collector disabled, not started
9: (8): ERROR: check_stun_auth: user self credentials are incorrect
9: (9): ERROR: check_stun_auth: user self credentials are incorrect
34: (9): ERROR: check_stun_auth: user self credentials are incorrect
35: (8): ERROR: check_stun_auth: user self credentials are incorrect
51: (9): ERROR: check_stun_auth: user self credentials are incorrect
52: (9): ERROR: check_stun_auth: user self credentials are incorrect
52: (9): ERROR: check_stun_auth: user self credentials are incorrect
52: (8): ERROR: check_stun_auth: user self credentials are incorrect
52: (9): ERROR: check_stun_auth: user self credentials are incorrect
53: (9): ERROR: check_stun_auth: user self credentials are incorrect
53: (9): ERROR: check_stun_auth: user self credentials are incorrect
53: (8): ERROR: check_stun_auth: user self credentials are incorrect
53: (9): ERROR: check_stun_auth: user self credentials are incorrect
57: (8): ERROR: check_stun_auth: user self credentials are incorrect
62: (8): ERROR: check_stun_auth: user self credentials are incorrect
63: (8): ERROR: check_stun_auth: user self credentials are incorrect
65: (8): ERROR: check_stun_auth: user self credentials are incorrect
65: (8): ERROR: check_stun_auth: user self credentials are incorrect
79: (9): ERROR: check_stun_auth: user self credentials are incorrect
79: (8): ERROR: check_stun_auth: user self credentials are incorrect
79: (9): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
80: (9): ERROR: check_stun_auth: user self credentials are incorrect
80: (9): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
[root@iZbp1imzcyvws0523mzrg4Z ~]# netbird status
Daemon version: 0.24.4
CLI version: 0.24.4
Management: Connected
Signal: Connected
FQDN: ss-pre1.netbird.selfhosted
NetBird IP: 10.255.248.87/22
Interface type: Userspace
Peers count: 10/22 Connected
C:\Users\vvv>netbird status
Daemon version: 0.25.2
CLI version: 0.25.2
Management: Connected
Signal: Connected
FQDN: qh.netbird.selfhosted
NetBird IP: 10.255.249.205/22
Interface type: Userspace
Peers count: 10/22 Connected
Hello @Horus-K, Thank you for reporting the issue. To assist us in diagnosing and resolving the problem, could you please share the following information:
The detailed Netbird status from both nodes using the command: netbird status --detail
Additionally, provide the firewall rules from the node where the attempted ping occurred. You can obtain this information with the command: sudo nft list ruleset.
[root@iZbp1imzcyvws0523mzrg4Z ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
CILIUM_INPUT all -- anywhere anywhere /* cilium-feeder: CILIUM_INPUT */
ACCEPT tcp -- anywhere anywhere tcp dpt:openvpn
ACCEPT udp -- anywhere iZbp1imzcyvws0523mzrg4Z udp dpt:domain
ACCEPT tcp -- anywhere iZbp1imzcyvws0523mzrg4Z tcp dpt:domain
KUBE-FIREWALL all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere tcp dpt:openvpn
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT all -- 10.255.248.0/22 anywhere
ACCEPT icmp -- anywhere anywhere icmp echo-request state NEW,RELATED,ESTABLISHED
ACCEPT all -- 10.255.248.0/22 anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
CILIUM_FORWARD all -- anywhere anywhere /* cilium-feeder: CILIUM_FORWARD */
KUBE-FORWARD all -- anywhere anywhere /* kubernetes forwarding rules */
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
CILIUM_OUTPUT all -- anywhere anywhere /* cilium-feeder: CILIUM_OUTPUT */
ACCEPT udp -- iZbp1imzcyvws0523mzrg4Z anywhere udp spt:domain
ACCEPT tcp -- iZbp1imzcyvws0523mzrg4Z anywhere tcp spt:domain
KUBE-FIREWALL all -- anywhere anywhere
ACCEPT all -- anywhere 10.255.248.0/22
ACCEPT icmp -- anywhere anywhere icmp echo-reply state RELATED,ESTABLISHED
ACCEPT all -- anywhere 10.255.248.0/22
Chain CILIUM_FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere /* cilium: any->cluster on cilium_host forward accept */
ACCEPT all -- anywhere anywhere /* cilium: cluster->any on cilium_host forward accept (nodeport) */
ACCEPT all -- anywhere anywhere /* cilium: cluster->any on lxc+ forward accept */
ACCEPT all -- anywhere anywhere /* cilium: cluster->any on cilium_net forward accept (nodeport) */
Chain CILIUM_INPUT (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere mark match 0x200/0xf00 /* cilium: ACCEPT for proxy traffic */
Chain CILIUM_OUTPUT (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere mark match 0xa00/0xfffffeff /* cilium: ACCEPT for proxy return traffic */
MARK all -- anywhere anywhere mark match ! 0xe00/0xf00 mark match ! 0xd00/0xf00 mark match ! 0xa00/0xe00 /* cilium: host->any mark as from host */ MARK xset 0xc00/0xf00
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP all -- !loopback/8 loopback/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
Chain KUBE-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT all -- anywhere anywhere /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination
[root@iZbp1imzcyvws0523mzrg4Z ~]# netbird status --detail Peers detail:
qh.netbird.selfhosted: NetBird IP: 10.255.249.205 Public key: Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx Last connection update: 2023-12-28 16:21:50
Daemon version: 0.25.2 CLI version: 0.25.2 Management: Connected to https://netbird.xxxx.cn:33073 Signal: Connected to http://netbird.xxxx.cn:10000 FQDN: ss-pre1.netbird.selfhosted NetBird IP: 10.255.248.87/22 Interface type: Userspace Peers count: 12/22 Connected
C:\Windows\system32>netbird status -d Peers detail: ss-pre1.netbird.selfhosted: NetBird IP: 10.255.248.87 Public key: xx Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx Last connection update: 2023-12-28 16:22:01
Daemon version: 0.25.2 CLI version: 0.25.2 Management: Connected to https://netbird.xxxx.cn:33073 Signal: Connected to http://netbird.xxxx.cn:10000 FQDN: qh.netbird.selfhosted NetBird IP: 10.255.249.205/22 Interface type: Userspace Peers count: 12/22 Connected
@Horus-K, we asked a few more questions via Slack; it might be better to troubleshoot there.
Nobody else will benefit from slack advice, google does not index it and slack free account does not show anything older than 3months
I have the exact same Problem: Mobile Devices can not connect to devices which are not directly accessable over the internet e.g. behind a firewall. The TURN server seems not to step in and it might be because of this error:
ERROR: check_stun_auth: user self credentials are incorrect
The username and password in management.json and turnserver.conf are the same. I checked them also within the containers. If there have been a solution for this problem here it might help me also.
I troubleshooted the problem and found out, that my problem was the coturn server which is also placed behind a NAT gateway with dynamic ip addresses. Therefor the external_ipparameter in /etc/turnserver.conf can not be filled but should be. I wrote a little script which updates the external_ip statement whenever the public address changed:
#!/bin/bash
# path to turnserver.conf
TURN_CONF="/etc/turnserver.conf"
# get external ip address
EXTERNAL_IP=$(curl -s ifconfig.me)
# read current ip address from turnserver.conf
CURRENT_IP=$(grep "^external-ip" "$TURN_CONF" | awk -F"=" '{print $2}')
# check if public and current ip address differ
if [ "$EXTERNAL_IP" != "$CURRENT_IP" ]; then
# set new public ip address
sed -i "s/^external-ip.*/external-ip=$EXTERNAL_IP/" "$TURN_CONF"
# restart coturn
systemctl restart coturn
echo "Coturn was restarted with a new external ip address: ($EXTERNAL_IP)" | logger
else
echo "The external ip address has not changed." | logger
fi
Whomever this may help.