netbird icon indicating copy to clipboard operation
netbird copied to clipboard

NetBird fails to forward traffic from Oracle Cloud VM to internal peer, works fine with Tailscale

Open fedeiglesias opened this issue 1 year ago • 4 comments

Describe the problem

I'm facing an issue where NetBird does not properly forward traffic from an Oracle Cloud VM to a peer in my internal network. I followed the exact same steps with Tailscale, and everything works fine under the same conditions. However, with NetBird, the traffic does not reach its destination.

I have an Oracle Cloud VM running Ubuntu (Ampere ARM64, 4 cores, 24 GB RAM) with a static public IP. On this VM, I installed NetBird from scratch, logged into my cloud account, and have a policy that allows all peers to connect to each other.

In my homelab, I have an LXC container running uptimekuma service on port 3001. The internal NetBird IP of this peer is 100.88.192.94. If I run a curl command from the Oracle VM, I can retrieve the HTML response from the NGINX service, so the internal connection between peers is working.

The issue arises when I try to forward the traffic coming to the VM on port 80 to the internal NetBird peer. I configured traffic forwarding using iptables, but it doesn't seem to work with NetBird, whereas with Tailscale, the traffic is correctly forwarded without issues.

To Reproduce

Steps to reproduce the behavior:

  1. Create an Oracle Cloud VM (Ampere ARM64, Ubuntu minimal).
  2. Install NetBird and set up a peer in the internal network (in this case, an LXC container running an NGINX service).
  3. On the Oracle VM, enable traffic forwarding and configure iptables to forward traffic on port 80 to the NetBird peer on port 3001.
  4. Try to access the web service through the Oracle Cloud public IP.

Expected behavior

I expected the incoming traffic on port 80 of the Oracle Cloud VM to be correctly forwarded to the internal NetBird peer, as it happens with Tailscale under the same conditions.

Are you using NetBird Cloud?

Yes, I am using NetBird Cloud.

NetBird version

0.28.7

NetBird status -dA output

NetBird status -dA output ubuntu@oracle:~$ netbird status -dA output Peers detail: iphone-fede.netbird.cloud: NetBird IP: 100.88.5.103 Public key: WKZknsTPHEZQREt75Kmaz/HV2jwcNehnpt0S91PKH1c= Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:57557 Last connection update: 22 minutes, 50 seconds ago Last WireGuard handshake: 1 minute, 48 seconds ago Transfer status (received/sent) 3.8 KiB/1.6 KiB Quantum resistance: false Routes: - Latency: 22.578577ms

oracle.netbird.cloud: NetBird IP: 100.88.61.146 Public key: PGT03R1EDb1cAf+uZymgvGYFXKHnnOP3/0ccKPInfBI= Status: Disconnected -- detail -- Connection type: Direct: false ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Routes: - Latency: 0s

4da8810ea346.netbird.cloud: NetBird IP: 100.88.110.131 Public key: +AawwZqEzKesGbPISiXgeU0yfTfmtkGKXGs0v7U152s= Status: Disconnected -- detail -- Connection type: Direct: false ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Routes: - Latency: 0s

macbook-fede.netbird.cloud: NetBird IP: 100.88.180.38 Public key: bQZuQnpveGtcU45nr7DTJtlWbhqi6O7rj/UwkNcB1iA= Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:57431 Last connection update: 22 minutes, 53 seconds ago Last WireGuard handshake: 2 minutes, 3 seconds ago Transfer status (received/sent) 3.6 KiB/1.2 KiB Quantum resistance: false Routes: - Latency: 14.533849ms

uptimekuma.netbird.cloud: NetBird IP: 100.88.192.94 Public key: hA3DLsvCah9Gz6YeWWWRGEBtCusBhadDopDOom0a2Qs= Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:58213 Last connection update: 22 minutes, 53 seconds ago Last WireGuard handshake: 2 minutes, 25 seconds ago Transfer status (received/sent) 3.0 KiB/2.1 KiB Quantum resistance: false Routes: - Latency: 20.015482ms

nginx.netbird.cloud: NetBird IP: 100.88.211.126 Public key: 9kpbqwyghDtjVPX92Ds0EF054+RzOMMw8/+efConO2I= Status: Disconnected -- detail -- Connection type: P2P Direct: false ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:57104 Last connection update: - Last WireGuard handshake: 1 minute, 48 seconds ago Transfer status (received/sent) 3.8 KiB/1.3 KiB Quantum resistance: false Routes: - Latency: 0s

pihole.netbird.cloud: NetBird IP: 100.88.212.19 Public key: gmSdBrNEI2j5fNwAWHRGyYhMAJUjTizb2HVqHEhq20Y= Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:57104 Last connection update: 22 minutes, 53 seconds ago Last WireGuard handshake: 1 minute, 48 seconds ago Transfer status (received/sent) 3.8 KiB/1.3 KiB Quantum resistance: false Routes: - Latency: 9.531658ms

OS: linux/arm64 Daemon version: 0.28.7 CLI version: 0.28.7 Management: Connected to https://api.netbird.io:443 Signal: Connected to https://signal.netbird.io:443 Relays: [stun:stun.netbird.io:5555] is Available [turns:turn.netbird.io:443?transport=tcp] is Available Nameservers: [100.88.212.19:53] for [local.anon-are26.domain] is Available FQDN: oracle-1.netbird.cloud NetBird IP: 100.88.223.175/16 Interface type: Kernel Quantum resistance: false Routes: - Peers count: 4/7 Connected

Do you face any client issues on desktop?

No desktop client is involved, only the Oracle VM client and the peer in my homelab (LXC).

Screenshots

CleanShot 2024-08-19 at 11  29 13@2x

Additional context

Here are the commands I used to configure port forwarding:

  • enable port forwarding sudo sed -i 's/^#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf sudo sysctl -p

  • add iptables rules sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 100.88.192.94:3001 sudo iptables -t nat -A POSTROUTING -j MASQUERADE

I have ensured that port 80 is open and accessible from the Oracle Cloud dashboard. As I mentioned before, this exact setup works fine with Tailscale, which makes me think this could either be a bug in NetBird or a misconfiguration on my side.

fedeiglesias avatar Aug 19 '24 21:08 fedeiglesias

Hi @fedeiglesias,

could you test if this build fixes the issue? https://github.com/netbirdio/netbird/actions/runs/10419607061/artifacts/1836477728

You can replace /usr/bin/netbird with the new binary on the oracle VM

lixmal avatar Aug 21 '24 09:08 lixmal

I'm having the same issue. As soon as the peer is connected, i cannot access the web server hosted on it via the public IP

@fedeiglesias did you find any solution to this?

ghaisasadvait avatar Aug 22 '24 10:08 ghaisasadvait

Same problem here with Netbird but working with Tailscale on same setup (Oracle Cloud VM and a VM on my local network)

raegedoc avatar Sep 25 '24 17:09 raegedoc

Hello @fedeiglesias,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

nazarewk avatar Apr 28 '25 15:04 nazarewk

@nazarewk i am using latest netbird and still facing this issue, as described in original post. This issue is replicable on Oracle Cloud VM

jainrahul0311 avatar Aug 12 '25 16:08 jainrahul0311

Root cause

On Oracle Cloud VMs, OCI injects its own iptables rules that REJECT or DROP inbound packets before UFW’s chains are evaluated. This means UFW “allow” rules for NetBird‑forwarded ports never match — traffic is killed before DNAT/MASQUERADE can happen.

Solution

Bypass UFW and insert explicit ACCEPT rules for the required ports above OCI’s default REJECT in the INPUT chain. For example:

bash
sudo iptables -I INPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -I INPUT 2 -p icmp -j ACCEPT
sudo iptables -I INPUT 3 -i lo -j ACCEPT
sudo iptables -I INPUT 4 -p udp --dport 3478 -j ACCEPT   # NetBird TURN
sudo iptables -I INPUT 5 -p tcp --dport 3000 -j ACCEPT   # Your service port
sudo iptables -I INPUT 6 -p tcp --dport 22 -j ACCEPT     # SSH

Then save the rules so they persist across reboots (example using iptables-persistent):

bash
sudo netfilter-persistent save

Result

  • NetBird traffic reaches the DNAT target.

  • Services become reachable from outside.

  • UFW can be removed entirely to avoid confusion.

gyrocoptic avatar Sep 22 '25 09:09 gyrocoptic