Exit node should not proxy traffic to Netbird peers - Windows 11

[alexcupertme]

Describe the problem

Platform: win11/x64 22H2 build 22621.4249 Netbird version 0.29.4

I think this maybe more than one bug so i'll try to show whole picture and chronology At first, we have encountered a problem that peer on Windows using exit node, cannot connect to SSH. It just gets stuck on message

debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: [email protected] MAC: <implicit                                                                                                                                                             > compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit                                                                                                                                                             > compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

There were also problems with SSL connection to peers. If we execute CURL on our peer webserver via HTTPS,

curl https://peer.website.mt.net --verbose
* Host peer.website.mt.net:443 was resolved.
* IPv6: (none)
* IPv4: 100.120.85.83
*   Trying 100.120.85.83:443...
* Connected to peer.website.mt.net (100.120.85.83) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* <stuck forever>

We tried to investigate and debug these issues and it was really tight because Netbird client is offering "plug and play" experience, which overall is great, but it misses usecases when something doesnt work and we need to configure Wireguard interface etc.

For example, we've found out that this issue happens when MTU of the client and router mismatched and client's is higher, and we cant change MTU via netbird config

We tried to change the MTU on user hosts in network interfaces directly. It helped, but all MTU settings disappeared after restarting the wt0 tunnel.

Also we noticed that problem happens only on peers with public IPs - they always have P2P connection and can be world accessible. So if we using Exit node and connect to the public server, exit node proxies traffic to the peer and our request gets stuck. Relayed servers, on other hand, dont have this problem because exit node not acting in this connection.

I think Wireguard running in Userspace may influence this problem -- Windows doesn't support Kernel mode. With Linux we didnt have problem

The workaround we ended with, creating systemd service dependent on netbird.service, which changes wt0 MTU to 1420. You don't have to change MTU on user machines, only exit nodes you use ase VPN:

root@jumphost:~# cat /etc/systemd/system/netbird-fix-mtu.service
[Unit]
Description=Start service when wt0 is up
Wants=network-online.target
After=network-online.target
BindsTo=sys-subsystem-net-devices-wt0.device
After=sys-subsystem-net-devices-wt0.device

[Service]
Type=oneshot
ExecStart=/sbin/ip link set dev wt0 mtu 1420
RemainAfterExit=yes

[Install]
WantedBy=sys-subsystem-net-devices-wt0.device

root@jumphost:~# cat /etc/udev/rules.d/99-net-interface-up.rules
ACTION=="add", SUBSYSTEM=="net", KERNEL=="wt0", TAG+="systemd", ENV{SYSTEMD_WANTS}="netbird-fix-mtu.service"
ACTION=="bind", SUBSYSTEM=="net", KERNEL=="wt0", TAG+="systemd", ENV{SYSTEMD_WANTS}="netbird-fix-mtu.service"

root@jumphost:~# sudo systemctl daemon-reload && sudo udevadm control --reload-rules && sudo systemctl enable netbird-fix-mtu.service

NetBird version

netbird version 0.29.4