netbird
netbird copied to clipboard
netbirdio
FORWARD don't work properly
Describe the problem Host A test to ping other host (B), don't work but if the node netbird on LAN A ping the host (B) is ok
The route is configured on netbird dashboard
Hi, I think it is useful to know more information, at least netbird version used as 0.24.4 have changes/fix relating forward https://github.com/netbirdio/netbird/pull/1305
Hello @barto95100 can you share the following details of your setup?
- is there are any custom configurations for the host A to access the IP of host B? any external routes, or custom firewall rules in any of the hosts?
- is NetBird A able to reach host B?
My responsable : is there are any custom configurations for the host A to access the IP of host B? any external routes, or custom firewall rules in any of the hosts?
YEs the host A sens paquet to routeur (gateway of host A) and on gateway (mikrotik) I havé a route to indicate to Subnet B -> NEtbird A, I start tcpdump on netbird 1 to view if paquet is received and it’s OK, and on netbird B nothing arrived from Netbird1
is NetBird A able to reach host B?
YEs netbird A access all host behind Netbird B
Hello @barto95100 ,
You need to add forward and maybe nat rules on netbird A:
sudo nft insert rule ip filter FORWARD ip saddr 10.150.10.49/32 accept
sudo nft insert rule ip filter FORWARD ip daddr 10.150.10.49/32 accept
sudo nft insert rule ip netbird netbird-rt-nat ip daddr 10.150.10.49/32 counter masquerade
Hi there,
I also have an issue with the current version (0.24.4) that looks similar to the problem described here. If I downgrade to 0.24.3 everything works fine again.
To my env: It is self-hosted and consists of 3 sites with a separate subnet behind Netbird (Netbird running on the gateway for each site). I have multiple routing rules so each site can reach systems in the subnets at the other sites (no masquerading).
It stopped working after upgrading to 0.24.4 from 0.24.3. Enabling masquerading in the routes also did not work. If I manually add the masquerading rule using iptables it starts working again.
From what I seen using tcpdump it looks like packets are not forwarded - e.g. (IPs are from the subnets of 2 sites - trying to ping a server from the subnet of site A (172.28.64.0/18) in the subnet of site B (172.28.0.0/18))
09:24:39.674826 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 245, length 64
09:24:40.681451 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 246, length 64
09:24:41.680920 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 247, length 64
09:24:42.682501 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 248, length 64
09:24:43.687602 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 249, length 64
09:24:44.689743 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 250, length 64
09:24:45.694963 IP 172.28.116.63 > 172.28.0.2: ICMP echo request, id 26408, seq 251, length 64
I already tried adding a few FORWARD ACCEPT rules with iptables incl. the nft rules from @mlsmaycon above without any success.
Hello @ydkn, can you share more information about your network setup and firewall rules added manually?
Is there any rules that you added before that is not being used in the new nftables chains?
Hi @mlsmaycon,
As I'm trying to use the setup without masquerading there are not much rules in place - see attached diagram for reference.
The configured routes also look fine - e.g. on 172.28.0.2:
172.28.64.0/18 via 172.29.254.112 dev wt0
172.28.128.0/18 via 172.29.254.112 dev wt0
172.29.252.0/22 dev wt0 proto kernel scope link src 172.29.254.112
I also did restarts of the gateway servers to avoid any rules that I added during testing interfere with the setup.
@ydkn thanks for sharing. Can you confirm that the gateways are also unable to connect to the IPs behind the other gateways?
@mlsmaycon I dug a little deeper and figured this out:
I had Netbird running not only on the gateways but also on most of the servers within the subnets of the site. Routing is setup to always use the local gateway for the other sites and also the routing rules in netbird only use the instance running on the gateway nodes to avoid routing conflicts. (That allowed my to have a fallback in case there is an issue with gateway gateway itself - internet access would still work)
The routes in Netbird are only assigned to the instances running on the gateway servers (there is also no route configured on the other systems running Netbird within the local subnet - so this seems to work)
If I shutdown Netbird on the other systems (and the local routing is used). I can reach systems behind the gateway from behind another gateways but not the gateway IP itself (local IP).
Example: I can ping 172.28.0.3 from 172.28.64.0/18 but NOT 172.28.0.2
Correction on my last post: Even with the Netbird agent running on other servers within the local subnet I can reach them from another site. I only have problems reaching things on the gateway server that do not use the Wiregurad/Netbird IP.
This would point into the direction that local forwarding does not work correctly.
If I ping the local IP of a remote gateway from another gateway it also works. I only can't reach the remote gateway IP from another subnet.
Example:
From 172.28.64.0/18 I can reach 172.28.0.0/18 except 172.28.0.2 (gateway) which is only reachable from 172.28.64.32 (gateway)
After setting up iptables, the network can be connected ， but，I want to know why the https protocol does not take effect
After setting up iptables, the network can be connected ， but，I want to know why the https protocol does not take effect
@Horus-K can you check configuring a larger MTU (e.g. 1420) for the nodes involved?
After setting up iptables, the network can be connected ， but，I want to know why the https protocol does not take effect
@Horus-K can you check configuring a larger MTU (e.g. 1420) for the nodes involved? in nbA nbB set
ifconfig wt0 mtu 1420doesn't work
ifconfig wt0 mtu 1420 in LAN 1
>curl https://xxxx.com/xx/awb
curl: (28) SSL/TLS connection timeout
@ydkn , would you join our Slack workspace to troubleshooting your issue?
@mlsmaycon I join your slack workspace.
Great @ydkn , pls reach out to me there. You can find me by searching for. mlsmaycon
ifconfig wt0 mtu 1420 in LAN 1
>curl https://xxxx.com/xx/awb curl: (28) SSL/TLS connection timeout
Can you run a tcpdump on the netbird server ss-pre-1? An example tcpdump command is:
sudo tcpdump -i any -n host 8.133.xxx.xxx and port 443
in ss-pre-1 [root@iZbp1imzcyvws0523mzrg4Z /]# tcpdump -i any -n host 101.132.xxx.xxx and port 443 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 19:48:29.835198 IP 10.255.250.88.63450 > 101.132.xxx.xxx.https: Flags [S], seq 3032769605, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 19:48:29.835253 IP 192.168.14.86.63450 > 101.132.xxx.xxx.https: Flags [S], seq 3032769605, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 19:48:29.844524 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [S.], seq 2521428729, ack 3032769606, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0 19:48:29.844556 IP 101.132.xxx.xxx.https > 10.255.250.88.63450: Flags [S.], seq 2521428729, ack 3032769606, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0 19:48:29.851658 IP 10.255.250.88.63450 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 1026, length 0 19:48:29.851686 IP 192.168.14.86.63450 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 1026, length 0 19:48:29.963545 IP 10.255.250.88.63450 > 101.132.xxx.xxx.https: Flags [P.], seq 1:201, ack 1, win 1026, length 200 19:48:29.963584 IP 192.168.14.86.63450 > 101.132.xxx.xxx.https: Flags [P.], seq 1:201, ack 1, win 1026, length 200 19:48:29.972736 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], ack 201, win 131, length 0 19:48:29.972762 IP 101.132.xxx.xxx.https > 10.255.250.88.63450: Flags [.], ack 201, win 131, length 0 19:48:29.976338 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [P.], seq 1:2921, ack 201, win 131, length 2920 19:48:29.976371 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [P.], seq 2921:3210, ack 201, win 131, length 289 19:48:29.976380 IP 101.132.xxx.xxx.https > 10.255.250.88.63450: Flags [P.], seq 2921:3210, ack 201, win 131, length 289 19:48:29.982601 IP 10.255.250.88.63450 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 1026, options [nop,nop,sack 1 {2921:3210}], length 0 19:48:29.982624 IP 192.168.14.86.63450 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 1026, options [nop,nop,sack 1 {2921:3210}], length 0 19:48:29.998697 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], seq 1:1461, ack 201, win 131, length 1460 19:48:30.219979 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], seq 1:1461, ack 201, win 131, length 1460 19:48:30.660986 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], seq 1:1461, ack 201, win 131, length 1460 19:48:31.588004 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], seq 1:1461, ack 201, win 131, length 1460 19:48:33.380978 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], seq 1:1461, ack 201, win 131, length 1460 19:48:36.900002 IP 101.132.xxx.xxx.https > 192.168.14.86.63450: Flags [.], seq 1:1461, ack 201, win 131, length 1460 @mlsmaycon
in openwrt curl https is ok
tcpdump in ss-pre-1
19:51:36.819965 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [S], seq 628007990, win 65535, options [mss 1420,sackOK,TS val 2786663374 ecr 0,nop,wscale 9], length 0
19:51:36.820013 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [S], seq 628007990, win 65535, options [mss 1420,sackOK,TS val 2786663374 ecr 0,nop,wscale 9], length 0
19:51:36.830044 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [S.], seq 119861510, ack 628007991, win 65535, options [mss 1460,sackOK,TS val 1003219675 ecr 2786663374,nop,wscale 9], length 0
19:51:36.830073 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [S.], seq 119861510, ack 628007991, win 65535, options [mss 1460,sackOK,TS val 1003219675 ecr 2786663374,nop,wscale 9], length 0
19:51:36.844752 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 128, options [nop,nop,TS val 2786663390 ecr 1003219675], length 0
19:51:36.844777 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 128, options [nop,nop,TS val 2786663390 ecr 1003219675], length 0
19:51:36.844783 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 1:518, ack 1, win 128, options [nop,nop,TS val 2786663394 ecr 1003219675], length 517
19:51:36.844787 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 1:518, ack 1, win 128, options [nop,nop,TS val 2786663394 ecr 1003219675], length 517
19:51:36.854761 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [.], ack 518, win 131, options [nop,nop,TS val 1003219700 ecr 2786663394], length 0
19:51:36.854783 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [.], ack 518, win 131, options [nop,nop,TS val 1003219700 ecr 2786663394], length 0
19:51:36.858645 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 1:2817, ack 518, win 131, options [nop,nop,TS val 1003219704 ecr 2786663394], length 2816
19:51:36.858660 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 1:2817, ack 518, win 131, options [nop,nop,TS val 1003219704 ecr 2786663394], length 2816
19:51:36.858664 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 2817:3354, ack 518, win 131, options [nop,nop,TS val 1003219704 ecr 2786663394], length 537
19:51:36.858668 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 2817:3354, ack 518, win 131, options [nop,nop,TS val 1003219704 ecr 2786663394], length 537
19:51:36.864864 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 131, options [nop,nop,TS val 2786663418 ecr 1003219700,nop,nop,sack 1 {2817:3354}], length 0
19:51:36.864885 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 1, win 131, options [nop,nop,TS val 2786663418 ecr 1003219700,nop,nop,sack 1 {2817:3354}], length 0
19:51:36.867820 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 1409, win 136, options [nop,nop,TS val 2786663421 ecr 1003219704,nop,nop,sack 1 {2817:3354}], length 0
19:51:36.867840 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 1409, win 136, options [nop,nop,TS val 2786663421 ecr 1003219704,nop,nop,sack 1 {2817:3354}], length 0
19:51:36.867849 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3354, win 142, options [nop,nop,TS val 2786663422 ecr 1003219704], length 0
19:51:36.867858 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3354, win 142, options [nop,nop,TS val 2786663422 ecr 1003219704], length 0
19:51:36.908384 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 518:598, ack 3354, win 142, options [nop,nop,TS val 2786663462 ecr 1003219704], length 80
19:51:36.908413 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 518:598, ack 3354, win 142, options [nop,nop,TS val 2786663462 ecr 1003219704], length 80
19:51:36.908715 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 598:684, ack 3354, win 142, options [nop,nop,TS val 2786663463 ecr 1003219704], length 86
19:51:36.908735 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 598:684, ack 3354, win 142, options [nop,nop,TS val 2786663463 ecr 1003219704], length 86
19:51:36.909394 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 684:756, ack 3354, win 142, options [nop,nop,TS val 2786663463 ecr 1003219704], length 72
19:51:36.909408 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 684:756, ack 3354, win 142, options [nop,nop,TS val 2786663463 ecr 1003219704], length 72
19:51:36.918335 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [.], ack 598, win 131, options [nop,nop,TS val 1003219764 ecr 2786663462], length 0
19:51:36.918362 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [.], ack 598, win 131, options [nop,nop,TS val 1003219764 ecr 2786663462], length 0
19:51:36.918597 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 3354:3433, ack 598, win 131, options [nop,nop,TS val 1003219764 ecr 2786663462], length 79
19:51:36.918614 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 3354:3433, ack 598, win 131, options [nop,nop,TS val 1003219764 ecr 2786663462], length 79
19:51:36.918627 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [.], ack 684, win 131, options [nop,nop,TS val 1003219764 ecr 2786663463], length 0
19:51:36.918635 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [.], ack 684, win 131, options [nop,nop,TS val 1003219764 ecr 2786663463], length 0
19:51:36.918657 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 3433:3512, ack 684, win 131, options [nop,nop,TS val 1003219764 ecr 2786663463], length 79
19:51:36.918662 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 3433:3512, ack 684, win 131, options [nop,nop,TS val 1003219764 ecr 2786663463], length 79
19:51:36.918718 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 3512:3583, ack 684, win 131, options [nop,nop,TS val 1003219764 ecr 2786663463], length 71
19:51:36.918726 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 3512:3583, ack 684, win 131, options [nop,nop,TS val 1003219764 ecr 2786663463], length 71
19:51:36.919344 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [.], ack 756, win 131, options [nop,nop,TS val 1003219765 ecr 2786663463], length 0
19:51:36.919354 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [.], ack 756, win 131, options [nop,nop,TS val 1003219765 ecr 2786663463], length 0
19:51:36.925638 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 3583:3849, ack 756, win 131, options [nop,nop,TS val 1003219771 ecr 2786663463], length 266
19:51:36.925657 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 3583:3849, ack 756, win 131, options [nop,nop,TS val 1003219771 ecr 2786663463], length 266
19:51:36.925662 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [P.], seq 3849:3880, ack 756, win 131, options [nop,nop,TS val 1003219771 ecr 2786663463], length 31
19:51:36.925667 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [P.], seq 3849:3880, ack 756, win 131, options [nop,nop,TS val 1003219771 ecr 2786663463], length 31
19:51:36.926070 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3583, win 142, options [nop,nop,TS val 2786663480 ecr 1003219764], length 0
19:51:36.926088 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3583, win 142, options [nop,nop,TS val 2786663480 ecr 1003219764], length 0
19:51:36.926093 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 756:787, ack 3583, win 142, options [nop,nop,TS val 2786663480 ecr 1003219764], length 31
19:51:36.926097 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [P.], seq 756:787, ack 3583, win 142, options [nop,nop,TS val 2786663480 ecr 1003219764], length 31
19:51:36.932362 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3880, win 147, options [nop,nop,TS val 2786663486 ecr 1003219771], length 0
19:51:36.932382 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3880, win 147, options [nop,nop,TS val 2786663486 ecr 1003219771], length 0
19:51:36.936022 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [.], ack 787, win 131, options [nop,nop,TS val 1003219781 ecr 2786663480], length 0
19:51:36.936050 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [.], ack 787, win 131, options [nop,nop,TS val 1003219781 ecr 2786663480], length 0
19:51:36.938828 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [F.], seq 787, ack 3880, win 147, options [nop,nop,TS val 2786663492 ecr 1003219771], length 0
19:51:36.938848 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [F.], seq 787, ack 3880, win 147, options [nop,nop,TS val 2786663492 ecr 1003219771], length 0
19:51:36.948983 IP 101.132.xxx.xxx.https > 192.168.14.86.38152: Flags [F.], seq 3880, ack 788, win 131, options [nop,nop,TS val 1003219794 ecr 2786663492], length 0
19:51:36.949015 IP 101.132.xxx.xxx.https > 10.255.250.88.38152: Flags [F.], seq 3880, ack 788, win 131, options [nop,nop,TS val 1003219794 ecr 2786663492], length 0
19:51:36.954874 IP 10.255.250.88.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3881, win 147, options [nop,nop,TS val 2786663509 ecr 1003219794], length 0
19:51:36.954902 IP 192.168.14.86.38152 > 101.132.xxx.xxx.https: Flags [.], ack 3881, win 147, options [nop,nop,TS val 2786663509 ecr 1003219794], length 0
Hello @barto95100 ,
You need to add forward and maybe nat rules on netbird A:
sudo nft insert rule ip filter FORWARD ip saddr 10.150.10.49/32 accept sudo nft insert rule ip filter FORWARD ip daddr 10.150.10.49/32 accept sudo nft insert rule ip netbird netbird-rt-nat ip daddr 10.150.10.49/32 counter masquerade
Don’t work for me … :(
Nos other problem, on mac os x netbird connect, after X hours not access behind netbird B it’s necessary to disconnect/Re-connect to work again…. :(
After setting up iptables, the network can be connected ， but，I want to know why the https protocol does not take effect
After setting up iptables, the network can be connected ， but，I want to know why the https protocol does not take effect
iptables -I FORWARD -o br-lan -j ACCEPT
iptables -I FORWARD -o wt0 -j ACCEPT
iptables -t nat -I POSTROUTING -o wt0 -j MASQUERADE
Thanks Brother, It works for me. 感谢老哥，我这能跑。
@jiangslee Have you tried https service to work?
很抱歉，你说的https service是啥？比如我访问另一端路由下的https://192.168.2.43 ，能正常访问到，所以我认为它能跑。
Sorry，I don’t know what https service you are talking about. In my case, when I access https://192.168.2.43 under the routing at the other end, I can access it normally.So I think it works for me.
@jiangslee I upgraded to the latest 25.4, and https can also be accessed. This problem has always existed in the previous 23.x version, and this version is very stable in my current testing.
It's resolved in version 0.25.1
@jiangslee I upgraded to the latest 25.4, and https can also be accessed. This problem has always existed in the previous 23.x version, and this version is very stable in my current testing.
所以这个问题已经被修复了吧。 So this issue have been fixed.
