netbirdio
Peer not accessible after assigning a new public ip to the peer
Describe the problem
I've setup two peers in DigitalOcean. The first peer hosts the management, dashboard, and signal containers. The second peer is a client peer. After assigning the second peer a new public ip address, other peers cannot connect to the second peer anymore. These peers are stuck in connecting state and network routes via the second peer do not work anymore.
Both peers are Linux (Ubuntu) virtual machines running in DigitalOcean (aka droplets). In DigitalOcean you can assign a static public reserved ip to a droplet to make the server for example suitable to be ip whitelisted.
Although a droplet has a static public ip address by default, recreating a droplet results in a new static public ip address. The old ip address is lost and cannot be reused. To always have the same ip address for a droplet also after destroying and creating a new one, a reserved ip address can be used to assign to a droplet. A reserved ip is an "owned" ip address.
Example:
- original public ip: A.A.A.A
- reserved public ip: B.B.B.B
Netbird status for peer with original public ip:
Peers detail:
oneplus7t-eea.netbird.selfhosted:
NetBird IP: 100.99.169.30/32
Public key: bTdFb8ukmy5ukjXe/MwJKdGp80CIz+E/m3l/ermnShA=
Status: Disconnected
-- detail --
Connection type: P2P
Direct: false
ICE candidate (Local/Remote): host/prflx
ICE candidate endpoints (Local/Remote): 10.110.0.7:51820/198.51.100.0:51820
Last connection update: 16 hours, 2 minutes ago
Last WireGuard handshake: 1 minute, 3 seconds ago
Transfer status (received/sent) 34.8 KiB/8.8 KiB
Quantum resistance: false
Routes: -
Latency: 0s
macbook-pro-2.netbird.selfhosted:
NetBird IP: 100.99.229.225
Public key: bUcXgQz0McQXxDDI0LVyq0KlVNkq6rQ0KElZ6kMTKQQ=
Status: Connected
-- detail --
Connection type: P2P
Direct: true
ICE candidate (Local/Remote): host/prflx
ICE candidate endpoints (Local/Remote): 10.110.0.7:51820/198.51.100.0:51820
Last connection update: 3 hours, 13 minutes ago
Last WireGuard handshake: 1 minute, 3 seconds ago
Transfer status (received/sent) 34.8 KiB/8.8 KiB
Quantum resistance: false
Routes: -
Latency: 25.250441ms
OS: linux/amd64
Daemon version: 0.28.3
CLI version: 0.28.3
Management: Connected to https://vpn.anon-hqBZ1.domain:443
Signal: Connected to https://vpn.anon-hqBZ1.domain:443
Relays:
[stun:vpn.anon-hqBZ1.domain:3478] is Available
[turn:vpn.anon-hqBZ1.domain:3478?transport=udp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: frank-vpn-peer.netbird.selfhosted
NetBird IP: 100.99.227.52/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 1/2 Connected
Netbird status with reserved ip address:
Peers detail:
oneplus7t-eea.netbird.selfhosted:
NetBird IP: 100.99.169.30/32
Public key: bTdFb8ukmy5ukjXe/MwJKdGp80CIz+E/m3l/ermnShA=
Status: Disconnected
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 37 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 0s
macbook-pro-2.netbird.selfhosted:
NetBird IP: 100.99.229.225
Public key: bUcXgQz0McQXxDDI0LVyq0KlVNkq6rQ0KElZ6kMTKQQ=
Status: Connecting
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 10 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 86.746397ms
OS: linux/amd64
Daemon version: 0.28.3
CLI version: 0.28.3
Management: Connected to https://vpn.anon-Kiafu.domain:443
Signal: Connected to https://vpn.anon-Kiafu.domain:443
Relays:
[stun:vpn.anon-Kiafu.domain:3478] is Available
[turn:vpn.anon-Kiafu.domain:3478?transport=udp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: frank-vpn-peer.netbird.selfhosted
NetBird IP: 100.99.227.52/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 0/2 Connected
I already contacted DigitalOcean for this issue but they said not to be able to help and suggested to raise an issue here.
To Reproduce
Steps to reproduce the behavior:
- Create a droplet with Ubuntu 24.04
- Create a peer with management, dashboard, and signal containers
- Create a second droplet with Ubuntu 24.04
- Create a client peer
- Create a network route via the second peer
- Check netbird status
- Assign reserved ip to second peer (https://docs.digitalocean.com/products/networking/reserved-ips/how-to/outbound-traffic/)
- Optionally reboot
- Check netbird status -> one peer is stuck in connecting status
Expected behavior
Second peer to be still connectable by other peers and network routes to function, when assigning a reserved ip address.
Are you using NetBird Cloud?
self-hosted
NetBird version
0.28.3
NetBird status -d output:
See above
Hello @tomasznguyen , Can you run sudo netbird service restart on the peer where you changed the network and then check the connection status again with netbird status -d?
Hello @bcmmbaga I ran the commands. The status is the same:
Peers detail:
oneplus7t-eea.netbird.selfhosted:
NetBird IP: 100.99.169.30/32
Public key: bTdFb8ukmy5ukjXe/MwJKdGp80CIz+E/m3l/ermnShA=
Status: Disconnected
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 23 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 0s
macbook-pro-2.netbird.selfhosted:
NetBird IP: 100.99.229.225
Public key: bUcXgQz0McQXxDDI0LVyq0KlVNkq6rQ0KElZ6kMTKQQ=
Status: Connecting
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 2 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 0s
OS: linux/amd64
Daemon version: 0.28.3
CLI version: 0.28.3
Management: Connected to https://vpn.anon-CCsQE.domain:443
Signal: Connected to https://vpn.anon-CCsQE.domain:443
Relays:
[stun:vpn.anon-CCsQE.domain:3478] is Available
[turn:vpn.anon-CCsQE.domain:3478?transport=udp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: frank-vpn-peer.netbird.selfhosted
NetBird IP: 100.99.227.52/16
Interface type: Kernel
Quantum resistance: false
Routes: admin.anon-CCsQE.domain, frank-postgres-production-read-only-do-user-10361637-0.c.db.anon-iXx2r.domain, frankenergie.anon-Fa9mQ.domain, frankenergie-test.anon-Fa9mQ.domain, lynxsaas-replica-01.database.anon-yl7UJ.domain, webclient.frankenergie.anon-VZp3x.domain
Peers count: 0/2 Connected
Let me know what info I can further provide.
Seems like the issue is not on the Linux peer but on this peer, macbook-pro-2.netbird.selfhosted. Could you please share the output of netbird status -d from it?
Output of macbook-pro-2.netbird.selfhosted:
Peers detail:
frank-vpn-peer.netbird.selfhosted:
NetBird IP: 100.99.227.52
Public key: BsJ584NKMQpvEc5L3reQRp1tloN+jkve8a1KfxFZS1Y=
Status: Connecting
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 8 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 0s
OS: darwin/arm64
Daemon version: 0.28.2
CLI version: 0.28.2
Management: Connected to https://vpn.anon-1ygIs.domain:443
Signal: Connected to https://vpn.anon-1ygIs.domain:443
Relays:
[stun:vpn.anon-1ygIs.domain:3478] is Available
[turn:vpn.anon-1ygIs.domain:3478?transport=udp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: macbook-pro-2.netbird.selfhosted
NetBird IP: 100.99.229.225/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 0/1 Connected
@bcmmbaga When connecting from an Android client (netbird version 0.0.24), the status is also "connecting":
Peers detail:
oneplus7t-eea.netbird.selfhosted:
NetBird IP: 100.99.169.30
Public key: bTdFb8ukmy5ukjXe/MwJKdGp80CIz+E/m3l/ermnShA=
Status: Connecting
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 10 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 0s
macbook-pro-2.netbird.selfhosted:
NetBird IP: 100.99.229.225
Public key: bUcXgQz0McQXxDDI0LVyq0KlVNkq6rQ0KElZ6kMTKQQ=
Status: Connecting
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Last connection update: 8 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Routes: -
Latency: 0s
OS: linux/amd64
Daemon version: 0.28.3
CLI version: 0.28.3
Management: Connected to https://vpn.anon-51RCE.domain:443
Signal: Connected to https://vpn.anon-51RCE.domain:443
Relays:
[stun:vpn.anon-51RCE.domain:3478] is Available
[turn:vpn.anon-51RCE.domain:3478?transport=udp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: frank-vpn-peer.netbird.selfhosted
NetBird IP: 100.99.227.52/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 0/2 Connected
Can you share the logs from the Linux and macOS nodes? You can run netbird debug -A for 1m to collect the logs
@bcmmbaga Any news? Anything I can do to help?
I will take a look today and let you know
Hi @bcmmbaga. I saw that a new version of netbird has been released. Just wanted to let you know that I upgraded netbird on all peers to test. The peer is still hanging in "connecting" state.
@tomasznguyen let’s validate the traffic with tcpdump. You can run the following command on the macOS client:
sudo tcpdump -i any -nn udp host <NEW DIGITAL OCEAN IP>
replace the command with the new digital ocean IP. I am interested in the UDP packets to/from this address on port 51820. We should see both-way traffic. Feel free to share some of the output via DM with me via Slack.
We are encountering a similar issue when trying to use Digital Ocean droplet as an exit node.
When using the default IPv4 address of the droplet, everything runs well. When sending all traffic through the reserved IP address all traffic gets relayed. It works, but slower then expected.
We setup everything according to the guide linked before: https://docs.digitalocean.com/products/networking/reserved-ips/how-to/outbound-traffic/
