netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Published 20 hours ago • verified publisher icon netbirdio

Inconsistent Traffic Routing with Netbird Client on Windows 11 Leading to 403 Errors due to Exit Node bypassing

Open rihards-simanovics opened this issue 4 months ago • 2 comments

Describe the problem

When using the Netbird client on Windows 11, traffic for a specific mixed use server IP can be routed and accepted by the server through the internet, but some traffic can only be routed and accepted through the Netbird exit node. If internet-routable traffic hits the IP first, the Netbird client does not later add the ruleset in the IP table to route all traffic through the VPN exit node, even if typically traffic though exit node succeeds in reaching the site. This results in a 403 error on pages that require traffic to be routed specifically through the exit node.

Context: we have a load balancer which routes traffic for both out our clients and some non-critical internal tools such as internal Wikis. problem is that when the load balancer gets hit with a standard internet traffic on domains that can accept that internet traffic the routing table doesn't get changed since, well, no need for routing, but then when we access the internal wiki pages, we get 403 as those sites are only accessible if the traffic comes from the known internal exit node. It seems like once Netbird client determines that IP can be reached by the internet without any problems on one domain, when another domain hits the same server but needs that exit node connection it doesn't change its mind? Basically, creating a scheduled task to do these fixes it, but it's annoying as we need to do it on all developer (Win 11) machines route add xxx.xxx.xxx.xxx mask 255.255.255.255 100.90.54.18 metric 5.

To Reproduce

Steps to reproduce the behaviour:

  1. Configure the Netbird client on Windows 11. On Management Server ensure that at least one exit node with static internet IP exists and Win 11 client is set to use it, take note of windows routing table using route print.
  2. Create a separate server with Nginx that has two domains configured: Say foo.com and bar.com. foo.com can accept all internet traffic, whereas bar.com can only be accessed using the IP of an Exit node, all other traffic should receive an error status 403.
    1. Connect PC with the client to Netbird network, and attempt to first visit bar.com.
      1. Observe that page loads
      2. Take note of the routing table in windows using route print.
    2. Disconnect then re-connect PC with the client to Netbird network, now attempt to first visit foo.com:
      1. Observe that page loads,
      2. Take note of the routing table in windows using route print
    3. Attempt to visit bar.com
      1. Should now fail with 403
      2. Observer the routing table route print.

Expected behavior

All traffic for the specified IP address should be routed through the VPN exit node, avoiding 403 errors on pages that require this routing.

Are you using NetBird Cloud?

No, Using self-hosted netbird 0.30.0

NetBird status -dA output:

PS C:\Users\RihardsSimanovics> netbird status -dA
Peers detail:
 gws-de-1-1.netbird.selfhosted:
  NetBird IP: 100.90.4.66
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.60.1:51820/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 20.8978ms

 gws-uk-7.netbird.selfhosted:
  NetBird IP: 100.90.10.42
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): [REDACTED]:3289/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 3.5148ms

 gws-rs-thinkpad-2.netbird.selfhosted:
  NetBird IP: 100.90.29.243
  Public key: [REDACTED]
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (connection won't work without a permissive mode)
  Routes: -
  Latency: 0s

 gws-uk-1-1.netbird.selfhosted:
  NetBird IP: 100.90.79.155
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): [REDACTED]:65471/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 3 seconds ago
  Last WireGuard handshake: 1 minute, 28 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 14.612ms

 gws-de-2-1.netbird.selfhosted:
  NetBird IP: 100.90.102.160
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 127.0.0.1:51820/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 21.5059ms

 gws-uk-2-1.netbird.selfhosted:
  NetBird IP: 100.90.116.141
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/host
  ICE candidate endpoints (Local/Remote): [REDACTED]:3289/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 29 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 25.4924ms

 gws-de-3-1.netbird.selfhosted:
  NetBird IP: 100.90.172.76
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.1.10:51820/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 56 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 18.5452ms

 gws-de-2-2.netbird.selfhosted:
  NetBird IP: 100.90.191.31
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 172.27.48.1:51820/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 22.0036ms

 gws-uk-3-1.netbird.selfhosted:
  NetBird IP: 100.90.214.145
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): [REDACTED]:3289/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 3.4671ms

 gws-uk-6-1.netbird.selfhosted:
  NetBird IP: 100.90.219.10
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): [REDACTED]:3289/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 43 seconds ago
  Transfer status (received/sent) 34.9 MiB/8.4 MiB
  Quantum resistance: true
  Routes: 0.0.0.0/0
  Latency: 7.2578ms

 gws-uk-4-1.netbird.selfhosted:
  NetBird IP: 100.90.230.4
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 127.0.0.1:51820/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 56 seconds ago
  Last WireGuard handshake: 1 minute, 31 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 2.846ms

 gws-uk-4-vpn-bridge-.netbird.selfhosted:
  NetBird IP: 100.90.245.18
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/host
  ICE candidate endpoints (Local/Remote): [REDACTED]:3289/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 52 seconds ago
  Transfer status (received/sent) 1.6 MiB/171.4 KiB
  Quantum resistance: true
  Routes: -
  Latency: 3.2511ms
  
gws-uk-5-1.netbird.selfhosted:
  NetBird IP: 100.90.249.209
  Public key: [REDACTED]
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 172.20.160.1:51820/[REDACTED]:51820
  Relay server address:
  Last connection update: 36 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 30 seconds ago
  Transfer status (received/sent) 1.6 KiB/5.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 22.2985ms
  
OS: windows/amd64
Daemon version: 0.30.0
CLI version: 0.30.0
Management: Connected to https://[REDACTED].domain:443
Signal: Connected to https://[REDACTED].domain:443
Relays:
  [stun:[REDACTED].domain:3478] is Available
  [turn:[REDACTED].domain:3478?transport=udp] is Available
Nameservers:
FQDN: gws-rs-main-pc-1.netbird.selfhosted
NetBird IP: 100.90.54.18/16
Interface type: Userspace
Quantum resistance: true
Routes: -
Peers count: 12/13 Connected

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Here are the routing table printouts before and after manually adding the route:

PS C:\Users\RihardsSimanovics> route print
===========================================================================
   redacted
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
   ... redacted prior list
   xxx.xxx.xxx.xxx  255.255.255.255      192.168.1.1     192.168.1.10     36
   ... redacted after list
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
redacted
===========================================================================
Persistent Routes:
  None

After manually adding the route:

route add xxx.xxx.xxx.xxx mask 255.255.255.255 100.90.54.18 metric 5

PS C:\Users\RihardsSimanovics> route print
===========================================================================
   redacted
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
   ... redacted prior list
   xxx.xxx.xxx.xxx  255.255.255.255      192.168.1.1     192.168.1.10     36
   xxx.xxx.xxx.xxx  255.255.255.255         On-link      100.90.54.18     10
   ...redacted after list
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
redacted
===========================================================================
Persistent Routes:
  None

rihards-simanovics avatar Oct 10 '24 06:10 rihards-simanovics