netbird no p2p between routers with public addresses

no p2p between routers with public addresses

Open oleg506 opened this issue 4 months ago • 0 comments

Describe the problem

Two routers based on openwrt, which have public addresses, cannot establish a p2p connection. Ports 51820, 51822 are open on both sides, RP Filter is disabled

Expected behavior

Direct p2p connection between nodes

Are you using NetBird Cloud?

yes

NetBird version

0.39.2

Is any other VPN software installed?

Debug output

root@OpenWrt:~# netbird status -dA
Peers detail:

 r2.netbird.cloud:
  NetBird IP: 100.115.140.180
  Public key: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://streamline-de-fra1-2.relay.netbird.io:443
  Last connection update: 7 minutes, 25 seconds ago
  Last WireGuard handshake: 1 minute, 3 seconds ago
  Transfer status (received/sent) 65.9 KiB/61.2 KiB
  Quantum resistance: false
  Networks: 192.168.100.0/24
  Latency: 0s

Events:
  [INFO] SYSTEM (8724df8c-dfd2-45f3-90fb-98b7b220685b)
    Message: Network map updated
    Time: 17 minutes, 53 seconds ago
OS: linux/arm64
Daemon version: 0.39.2
CLI version: 0.39.2
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays: 
  [stun:stun.netbird.io:443] is Available
  [stun:stun.netbird.io:5555] is Available
  [turns:turn.netbird.io:443?transport=tcp] is Available
  [rels://streamline-fi-hel1-0.relay.netbird.io:443] is Available
Nameservers: 
FQDN: home.netbird.cloud
NetBird IP: 100.115.176.158/16
Interface type: Kernel
Quantum resistance: false
Networks: 172.20.1.0/24, 172.20.1.20/32, 192.168.10.0/24, 192.168.10.5/32, 192.168.15.0/24
Forwarding rules: 0
Peers count: 1/3 Connected

debug log.

1 side

2025-06-26T14:42:53Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.39.2 and with remote WireGuard listen port 51820
2025-06-26T14:42:53Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/handshaker.go:79: wait for remote offer confirmation
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:81: OnNewOffer for ICE
2025-06-26T14:42:53Z DEBG relay/client/manager.go:146: open peer connection via foreign server: rels://streamline-de-fra1-2.relay.netbird.io:443
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_relay.go:68: handled offer by reusing existing relay connection
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:97: recreate ICE agent
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:108: gather candidates
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:118: turn agent dial
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:211: ICE ConnectionState has changed to Checking
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:299: discovered local candidate udp4 host 109.200.226.*:51822
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:161: OnRemoteCandidate from peer fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo= -> udp4 host 109.122.5.*:51820
2025-06-26T14:42:53Z DEBG client/iface/bind/udp_mux.go:363: ICE: registered 109.122.5.*:51820 for XXqRsdUbEeoxRMeO
2025-06-26T14:42:53Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:161: OnRemoteCandidate from peer fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo= -> udp4 srflx 109.122.5.*:51820 related 0.0.0.0:51820
2025-06-26T14:42:56Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/conn.go:293: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2025-06-26T14:42:56Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/handshaker.go:170: sending answer
2025-06-26T14:42:56Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.39.2 and with remote WireGuard listen port 51820
2025-06-26T14:42:56Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/handshaker.go:79: wait for remote offer confirmation
2025-06-26T14:42:56Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:81: OnNewOffer for ICE
2025-06-26T14:42:56Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_ice.go:85: agent already exists, skipping the offer
2025-06-26T14:42:56Z DEBG relay/client/manager.go:146: open peer connection via foreign server: rels://streamline-de-fra1-2.relay.netbird.io:443
2025-06-26T14:42:56Z DEBG [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/worker_relay.go:68: handled offer by reusing existing relay connection
2025-06-26T14:42:58Z INFO [peer: fnQm9LNBAyUC0X9YGAVX5uStagT384cXXU/2ws1Hvzo=] client/internal/peer/conn.go:293: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected

2 side

025-06-26T14:52:42Z INFO [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/handshaker.go:91: received connection confirmation, running version 0.39.2 and with remote WireGuard listen port 51822
2025-06-26T14:52:42Z INFO [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/handshaker.go:79: wait for remote offer confirmation
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:81: OnNewOffer for ICE
2025-06-26T14:52:42Z DEBG relay/client/manager.go:143: open peer connection via permanent server: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_relay.go:68: handled offer by reusing existing relay connection
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:97: recreate ICE agent
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:108: gather candidates
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:118: turn agent dial
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:211: ICE ConnectionState has changed to Checking
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:299: discovered local candidate udp4 host 109.122.5.*:51820
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:299: discovered local candidate udp4 srflx 109.122.5.*:51820 related 0.0.0.0:51820
2025-06-26T14:52:42Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:161: OnRemoteCandidate from peer HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E= -> udp4 host 109.200.226.*:51822
2025-06-26T14:52:42Z DEBG client/iface/bind/udp_mux.go:363: ICE: registered 109.200.226.*:51822 for cQijsRdfueBBkWPu
2025-06-26T14:52:42Z DEBG client/iface/bind/udp_mux.go:363: ICE: registered 109.200.226.*:51822 for cQijsRdfueBBkWPuturns:turn.netbird.io:443?transport=tcp
2025-06-26T14:52:54Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:211: ICE ConnectionState has changed to Failed
2025-06-26T14:52:54Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:121: failed to dial the remote peer: connecting canceled by caller
2025-06-26T14:52:54Z DEBG [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/worker_ice.go:211: ICE ConnectionState has changed to Closed
2025-06-26T14:53:07Z INFO [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/conn.go:548: send offer to peer
2025-06-26T14:53:07Z INFO [peer: HK6ZUySZdl3BxXhhv4GlboMluyejzK7ZLszqFRYQ+3E=] client/internal/peer/conn.go:264: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected

The logs show that ICE detects the external address from both sides and tries to establish a connection, but for some reason it fails. At the same time, I tried to completely disable filtering at the entrance to the firewall. The tcpdump output also shows that requests come and go

tcpdump -i wan -nn port 51820 or port 5182

18:53:36.785500 IP 109.200.226.*.51822 > 80.69.173.91.443: UDP, length 20
18:53:36.787773 IP 109.200.226.*.51822 > 94.237.30.53.5555: UDP, length 20
18:53:36.787954 IP 109.200.226.*.51822 > 94.237.30.53.443: UDP, length 20
18:53:36.830855 IP 109.122.5.*.51820 > 109.200.226.*.51822: UDP, length 112
18:53:36.831520 IP 109.200.226.*.51822 > 109.122.5.*.51820: UDP, length 64
18:53:36.831868 IP 109.200.226.*.51822 > 109.122.5.*.51820: UDP, length 112
18:53:36.832* IP 109.200.226.*.51822 > 109.122.5.*.51820: UDP, length 112
18:53:36.876207 IP 109.122.5.*.51820 > 109.200.226.*.51822: UDP, length 112
18:53:36.876207 IP 109.122.5.*.51820 > 109.200.226.*.51822: UDP, length 11

Jun 26 '25 18:06 oleg506

netbird netbird copied to clipboard

no p2p between routers with public addresses

netbird
netbird copied to clipboard