netbirdio
Netbird >0.40 disconnects after some time.
Now I'm running on version 0.41.3 but from version 0.36 netbird routing peer disconnects from server when in inactive state and for some reason reporting different client versions in logs.
2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 2, RemoteOffer: 3, RemoteAnswer: 2, RemoteCandidate: 8, P2PConnected: 1, SwitchToRelay: 0, WGCheckSuccess: 0, RelayConnected: 0, LocalProxies: 0 2025-04-17T13:36:53Z INFO [peer: o] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 2, RemoteOffer: 0, RemoteAnswer: 0, RemoteCandidate: 0, P2PConnected: 0, SwitchToRelay: 0, WGCheckSuccess: 0, RelayConnected: 0, LocalProxies: 0 2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/conn.go:430: ICE disconnected, do not switch to Relay. Reset priority to: None 2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/conn.go:557: send offer to peer 2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/conn.go:272: OnRemoteAnswer, priority: None, status ICE: Disconnected, status relay: Disconnected 2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.40.1 and with remote WireGuard listen port 51820 2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:36:53Z INFO [peer: y] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:36:54Z INFO [peer: y] client/internal/peer/conn.go:344: set ICE to active connection 2025-04-17T13:36:54Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: 173.179.108.223:51820, endpoint port: :20 2025-04-17T13:38:12Z INFO [peer: x] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:38:12Z INFO [peer: x] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:38:12Z INFO [peer: x] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.36.6 and with remote WireGuard listen port 51820 2025-04-17T13:38:12Z INFO [peer: x] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:38:12Z INFO [peer: x] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:38:16Z INFO [peer: x] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:38:16Z INFO [peer: x] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:38:16Z INFO [peer: x] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.36.6 and with remote WireGuard listen port 51820 2025-04-17T13:38:16Z INFO [peer: x] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:38:16Z INFO [peer: x] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:38:17Z INFO [peer: x] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:38:17Z INFO [peer: x] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:38:17Z INFO [peer: x] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.36.6 and with remote WireGuard listen port 51820 2025-04-17T13:38:17Z INFO [peer: x] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:38:17Z INFO [peer: x] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:38:19Z INFO [peer: z] client/internal/peer/conn.go:430: ICE disconnected, do not switch to Relay. Reset priority to: None 2025-04-17T13:38:19Z INFO [peer: z] client/internal/peer/conn.go:557: send offer to peer 2025-04-17T13:38:20Z INFO [peer: x] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:38:20Z INFO [peer: x] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:38:20Z INFO [peer: x] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.36.6 and with remote WireGuard listen port 51820 2025-04-17T13:38:20Z INFO [peer: x] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:38:20Z INFO [peer: x] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:38:26Z INFO [peer: x] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:38:26Z INFO [peer: x] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:38:26Z INFO [peer: x] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.36.6 and with remote WireGuard listen port 51820 2025-04-17T13:38:26Z INFO [peer: x] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:38:26Z INFO [peer: x] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:38:27Z INFO [peer: x] client/internal/peer/conn.go:344: set ICE to active connection 2025-04-17T13:39:02Z INFO [peer: p] client/internal/peer/conn.go:430: ICE disconnected, do not switch to Relay. Reset priority to: None 2025-04-17T13:39:02Z INFO [peer: p] client/internal/peer/conn.go:557: send offer to peer 2025-04-17T13:39:08Z INFO [peer: p] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:39:08Z INFO [peer: p] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:39:08Z INFO [peer: p] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.39.2 and with remote WireGuard listen port 51820 2025-04-17T13:39:08Z INFO [peer: p] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:39:08Z INFO [peer: p] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:39:11Z INFO [peer: p] client/internal/peer/conn.go:301: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2025-04-17T13:39:11Z INFO [peer: p] client/internal/peer/handshaker.go:170: sending answer 2025-04-17T13:39:11Z INFO [peer: p] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.39.2 and with remote WireGuard listen port 51820 2025-04-17T13:39:11Z INFO [peer: p] client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-04-17T13:39:11Z INFO [peer: p] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer 2025-04-17T13:39:12Z INFO [peer: p] client/internal/peer/conn.go:344: set ICE to active connection
I'm also getting reports of these disconnections. For info's sake, reported users were on client version 0.39.2. After upgrading to 0.43.0, disconnections have stopped.
**Edit: Btw, I think you mean <0.40 (less than) disconnects, right @xromansx ?
Has anyone fixed the problem yet?
I have the same problem, also still in version 0.46.0
2025-06-06T08:32:47+02:00 INFO [peer: $randomstring] client/internal/peer/conn.go:311: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2025-06-06T08:32:47+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:168: sending answer
2025-06-06T08:32:47+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:89: received connection confirmation, running version 0.46.0 and with remote WireGuard listen port 51820
2025-06-06T08:32:47+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:77: wait for remote offer confirmation
2025-06-06T08:32:49+02:00 INFO [peer: $randomstring] client/internal/peer/conn.go:282: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected
2025-06-06T08:32:49+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:89: received connection confirmation, running version 0.46.0 and with remote WireGuard listen port 51820
2025-06-06T08:32:49+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:77: wait for remote offer confirmation
2025-06-06T08:32:51+02:00 INFO [peer: $randomstring] client/internal/peer/conn.go:311: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2025-06-06T08:32:51+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:168: sending answer
2025-06-06T08:32:51+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:89: received connection confirmation, running version 0.46.0 and with remote WireGuard listen port 51820
2025-06-06T08:32:51+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:77: wait for remote offer confirmation
2025-06-06T08:32:53+02:00 INFO [peer: $randomstring] client/internal/peer/conn.go:311: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2025-06-06T08:32:53+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:168: sending answer
2025-06-06T08:32:53+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:89: received connection confirmation, running version 0.46.0 and with remote WireGuard listen port 51820
2025-06-06T08:32:53+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:77: wait for remote offer confirmation
2025-06-06T08:32:56+02:00 INFO [peer: $randomstring] client/internal/peer/conn.go:311: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2025-06-06T08:32:56+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:168: sending answer
2025-06-06T08:32:56+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:89: received connection confirmation, running version 0.46.0 and with remote WireGuard listen port 51820
2025-06-06T08:32:56+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:77: wait for remote offer confirmation
2025-06-06T08:33:02+02:00 INFO [peer: $randomstring] client/internal/peer/conn.go:311: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2025-06-06T08:33:02+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:168: sending answer
2025-06-06T08:33:02+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:89: received connection confirmation, running version 0.46.0 and with remote WireGuard listen port 51820
2025-06-06T08:33:02+02:00 INFO [peer: $randomstring] client/internal/peer/handshaker.go:77: wait for remote offer confirmation
Unfortunately, I can't find a community group at the moment. I took the liberty of creating an unofficial community room on Matrix. Anyone who would like to join is welcome to do so. https://matrix.to/#/#netbird-community:techniverse.net
Hello Folks, if you still facing this issue, can you please share a debug bundle?
https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle-uploads
Hello Folks, if you still facing this issue, can you please share a debug bundle?
Thank you very much for your feedback.
➜ ~ netbird debug bundle --system-info --upload-bundle
Local file:
/tmp/netbird.debug.3379241334.zip
Upload file key:
2386d2e4a9def9c7a783b254006addcd4b2251c1f4865d1cd237d84919dd3707/2fe0acee-6336-4943-adb3-76fda3184f90
Do you need anything else from me?
actually, if it happens every few minutes, can you run this command instead?:
netbird debug for 1m --system-info --upload-bundle -A
Adjust the 1m if it takes more than 1 minute for the peers to reconnect.
Also, can you confirm the peer NetBird IP that is reconnecting? that will help us focus on specific log events
The current issue is that the Netbird client with the IP address 100.83.12.99 (vps02-docker01) should establish a connection to my server network. This usually happens by using a gateway. The gateway has the IP address 100.83.229.208 (vhost02) in the Netbird network.
Initially, it helped if I briefly deactivated the resources and then reactivated them. Unfortunately, this no longer works today.
➜ ~ netbird debug for 10m --system-info --upload-bundle -A
Log level set to trace.
Netbird down
Netbird up
Remaining time: 00:00:01
Duration completed
Creating debug bundle...
Log level restored to INFO
Local file:
/tmp/netbird.debug.351188597.zip
Upload file key:
2386d2e4a9def9c7a783b254006addcd4b2251c1f4865d1cd237d84919dd3707/0ffcd66c-f1db-4684-80a4-6cc5937b1be6
I have just deactivated and reactivated a resource (server network) while creating the logs I hope all this helps you. If there is anything else I can do, please let me know.
I’d like to mention that we’re experiencing a similar issue: after some time—typically more than a month—the Netbird service needs to be restarted for new peers to successfully connect. Existing connections continue to work fine, but newly connecting peers fail to establish a connection with the routers. We’re running the latest version (59.x). I have another setup running an older version of Netbird (0.32.x), and I’ve never encountered this issue there.
When peers can not connect, it seems that they are not able to get the ice candidate endpoints:
ip-10-110-1-130-38-228.netbird.selfhosted:
NetBird IP: 100.127.38.228
Status: Connecting
-- detail --
Connection type: -
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
Relay server address:
Last connection update: 3 minutes, 31 seconds ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B
Quantum resistance: false
Networks: -
Latency: 0s
Is netbird retrying to get again ice candidates when it is not able to find them in P2P connections ? A netbird service restart solves the issue.
Would it be possible to add a new feature in netbird dashboard, to trigger a netbird service restart of a peer?
