netbird
netbird copied to clipboard
netbirdio
Allow Route to FreeBSD client
I've created a quick and dirty FreeBSD port and an OPNSense plugin for the netbird client. This works quite well but I had to patch it. To prevent netbird from messing with the routing table of the firewall I've set NB_DISABLE_CUSTOM_ROUTING to true. Then I tried to enable a route to the network behind the firewall. This failed because the management service only allows routes to Linux clients. To fix this I made the client pretend to be running on linux. e.g. I've patched info_freebsd.go. Now our firewall works as netbird gateway for our LAN. Are there any plans to allow routes to FreeBSD clients?
Really Cool, @moserpjm. Would you share the port and the code changes you've done? We plan to add support to PFsense and OPNSense soon, but we are a bit short on capacity and any community help is welcome.
Due to the fact that we're a JVM shop I'm not an expert on BSD ports. :D I have two repos on our Bitbucket server. One with a clone of the FreeBSD ports tree plus the folder security/netbird and a second with a clone of the OPNSense plugins GH repo plus the folder net/netbird. The feature set of the plugin is service control and creation of a CARP hook script to execute netbird up/down on change of the master node. What would be the best way to make this source accessible for you? Unfortuately I don't have the time to try to get it into the official ports / plugin trees.
I'll cleanup the code a little bit, push it to GitHub and then invite you.
I've forked the repos on GH and added my changes. Build works for me on the newest OPNSense version. Netbird has to be configured manually via the CLI. In case the CARP feature should be used auto connect must be disabled. Without CARP it has to be enabled.
Netbird port https://github.com/moserpjm/freebsd-ports
Build instruction:
cd net/netbird
make makesum
make package
OPNSense plugin https://github.com/moserpjm/opnsense-plugins
Build instruction:
cd net/netbird
make package
Great to see the community getting involved and being able to contribute towards this! Well done @moserpjm :) This is a feature that a lot of us are waiting for (reduces infrastructure requirements for self-hosting netbird and lets us re-use existing equipment)
@moserpjm thanks for this, built and tested on pfsense arm and opnsense x86-64, works on both, including egress.
@robdeweese great to hear. I'm currently reworking the CARP support. The current implementation regenerates the hook script every time the settings are saved. Unfortunately the execute flag of the script gets lost in this process. After digging through other plugins I'm now reimplementing it as a PHP script which dynamically fetches the settings. Looks like that's the preferred way.
Is there an OPNSense repo I can use to install these? I'd love to test them out.
Is there an OPNSense repo I can use to install these? I'd love to test them out.
I've put the packages ive built here https://nhd.cx/w6we3 but I only build the client, not the web interface. They will run on pfsense and opnsense
but they are very easy to build yourself as well, @moserpjm provided instructions above
Just some quick updates:
I found out hat OPNSense is maintaining a fork of the freebsd-ports repo with tags of all releases in it. So I've rebased on that. https://github.com/moserpjm/opnsense-ports -> Current branch is 24.1.10-netbird-develop.
There are new patches included:
- sets the host manager to noop. This prevents netbird from even trying to update the resolv.conf file.
- stets the reveived routes proto to nil to avoid the warning that it received a route but shouldnt as it's FreeBSD. This should alsow avoid any routing table incidents when nebird starts supporting FreeBSD routes in the future.
It would be really nice to have environment variables for those settings in netbird to get rid of the patches.
The plugin now has some new features:
- New status page with filter/sortable host table
- Manual netbird up/down control on the status page.
- Form to do the initial netbird up -k .... (only -k -m and -n available via GUI atm)
- New php based CARP syshook
- Checkboxes to enable rosenpass / rosenpass permissive
- Ability to set the wireguard port.
My current working branch is "new-status".
We're currently testing it with our firewalls at the office. A repo for internal use is already in place. I'll try to get a cheap VPS to let you try my builds.
Is there an OPNSense repo I can use to install these? I'd love to test them out.
If you're brave enough you can try the build in my RC repo.
- fetch -o /usr/local/etc/pkg/repos/netbird-rc-241.conf https://os-pkg.pjm.co.at/netbird-rc-241.conf
- pkg update
- Install os-netbird (It should show up under VPN)
- Add a firewall rule to open the desired WG port
- Change the settings to your liking and enable it.
- Use the setup function on the settings page or do manual netbird up with the desired options.
- Assign the wt0 interface
- Enable the interface (Don't forget to check "Prevent interface removal")
- If you enable the CARP functionality you have to press the "Set UP" button on the connection status page of the MASTER otherwise both nodes are down until a CARP event happens.
I've uploaded 0.28.5 builds to the PR repo. Please stop the service before installing the update. This update fixes problem of netbird not stating if it crashed before and didn't delete it's wt0 tun device. Also syslog output is now enabled. The OS plugin now contains the appropriate syslog filter and a link to the log viewer.
I've also restructured the repos. There are now development branches for OS 24.1 and 24.7 in both repos. For future builds I'll tag them in GIT with the port/plugin version.
24.7 versions work fine on the latest R2. My only problem is the maintainance nightmare of two versions. :D
@moserpjm
~~nice work! I compiled both packages from your github repos on 24.7_5 myself and it is up and running. I do see the opnsense online on netbird admin page. I can ping opnsense and the networks behind, but I don't get any traffic through. I assigned wt0 and added a firewall rule from any to any. Without this rule, pings are also not possible. The status page of the plugin shows all peers as connected.~~
~~Am I missing something?~~
I think I do have some glitches in my config. This plugin seems to work without any problem and the problems came (unexpected) out of my system.
~~Am I missing something?~~
~~I was not clear in my previous post. I only tried traffic to port 443 on several web services behind opnsense and opnsense. Also opnsense is configured as routing peer in netbird. I now found 2 rules in NAT.~~
~~If I disable the 443 rule, traffic on 443 to clients behind opnsense is working.~~
Just opnsense itself is complaining about potential DNS rebind attack, if connecting through netbird DNS. This behaviour is fine and default to OPNsense.
~~Using e. g. LAN IP of opnsense, everything is working as expected.~~
I think I do have some glitches in my config. This plugin seems to work without any problem and the problems came (unexpected) out of my system.
@Hobby-Student nice to hear that it works for you. Just created the RC repo for 24.7: fetch -o /usr/local/etc/pkg/repos/netbird-rc-247.conf https://os-pkg.pjm.co.at/netbird-rc-247.conf
@moserpjm
To prevent netbird from messing with the routing table of the firewall I've set NB_DISABLE_CUSTOM_ROUTING to true.
with the netbird port wt0 is added to the system and we enable the netbird interface within opnsense. For me it looks like you then need to manage the firewall rules for the netbird interface. If netbird would add routes in the background, those routes wouldn't affect opnsense in a negative way?
As far as I can see, tailscale is doing it the same way. Just doing stuff in background and opnsense can handle this through firewall rules on the interface.
I'll do some tests.
Yes you have to manage the rules on your own. If you configure a network route via a node in Netbird firewall rules don't apply anyway. Don't get me wrong but no sane sysadmin would trust the firewall mechanism of some third party VPN tool on a firewall. Client maybe but not on a firewall. Like I said I'm also no big fan of letting netbird change the routing table. Wrong entries, like overriding the standard gateway, could bring the whole firewall down. But hey that's just my option. The beauty of open source is that you can change it however you like. 😉
[...] no sane sysadmin would trust the firewall mechanism of some third party VPN tool on a firewall.[...]
I agree, but I want to give the possibility a chance and take a look at how things are handled this way. I did try tailscale the other day and I think they use this approach. Also the wireguard plugin of opnsense is adding the routes from the config file and I'm not experienced enough to see, if adding routes by netbird would act the same way. If so, a sane admin could whitelist the needed netbird IPs without relying on the netbird created peers (and routes) as a whole 😉
Of course, I could be totally wrong 😄
little Update:
I compiled the ports version of @moserpjm with some modifications to patch-opnsense. I removed everything except the changes for operating system and dns. I then added a static route to opnsense with route -n add -inet 10.0.0.16/28 -interface wt0. netbird routing is configured with a linux machine providing the route to 10.0.0.16/28 (masquerade) and opnsense is allowed to use it. But it's not working. For other peers, the route is working as expected.
Is netbird client not able to use the routes because it's not implemented for FreeBSD and therefore static routes are not working?
Pro tip #1: Don't mess around with the shell. OPNsense has it's own configuration system. If you fiddle around with the CLI it will end in some undefined state. I'd reboot it first before you do anything else.
I haven't tried what you try to accomplish myself so this is just a lucky guess. Try adding a gatway with the netbird ip of your linux machine on the netbird interface. Then add a route via the gateway to 10.0.0.16/28. Yes all via the webui. ;) This is what netbird would do automatically if it would know how on BSD.
Pro tip #1: Don't mess around with the shell. OPNsense has it's own configuration system. If you fiddle around with the CLI it will end in some undefined state. I'd reboot it first before you do anything else.
I haven't tried what you try to accomplish myself so this is just a lucky guess. Try adding a gatway with the netbird ip of your linux machine on the netbird interface. Then add a route via the gateway to 10.0.0.16/28. Yes all via the webui. ;) This is what netbird would do automatically if it would know how on BSD.
I tried it first through the web UI. It failed. Then I did what wireguard does (or how I think it does), just add the route on the interface without a gateway. Perhaps I missed something and I will try again as soon as possible.
Mystery solved (perhaps): netbird routes list -> No routes available
@mlsmaycon Am I right, that routing is not working because it's not implemented yet for FreeBSD? Or should routing work if I add it manually as a static route - with or without the routing peer as gateway?
removed netbird from opnsense, rebooted, deleted the peer in netbird management, compiled netbird v0.28.7 and just used the mod by @moserpjm to identify the FreeBSD as linux -> It works!
Don't know exactly why it wasn't before, but I think there was something wrong on my side. Also the rules of opnsense are working with the netbird routing.
I'll test this some more and report back.
Deployed it on 3 opnsense boxes and configured routes via netbird management. Every firewall rule inside OPNsense is respected and withtout any rule on netbird interface, traffic is blocked (default). So far I can't see any showstoppers in this small test environment. Routing between the OPNsense boxes and access to all networks (with configured firewall and outbound NAT rules) are working as intended.
I'll have a look at the source. Looks like Nerbird needs those routes set via the management service. Makes somehow sense. Wireguard connections have allowed CIDRs configured on them. Nerbird is setting up the connections so it has to do the configuration. You also removed the patches which disable setting DNS servers. They are not really necessary. Nerbird fails setting them on OPNsense anyway (no resolve.conf in the locations it searches for one) but I wanted to get rid of the errors in the log file. ;) I'll look into patching in an environment variable to enable or disable routes. Like the accept routes option in Tailscale. People who need them can then enable it via the UI and people like me can disable it. We only use Nerbird on OPNsense to make our office network available to clients. No outgoing routes needed. Better safe than sorry. I managed to push a wrong route once via Zerotier which brought our whole network down...
I have added FreeBSD support to Netbird client, and from what I know, routing on FreeBSD untested, and currently 3 routing related e2e tests completely disabled for FreeBSD. Take a look this issue for details: https://github.com/netbirdio/netbird/issues/2386 But I have not found any critical problems with routing on FreeBSD so far in the current version, but that doesn't mean they don't exist. It would be great if you could share any routing or DNS-related problems and how to reproduce them on FreeBSD as a separate issue. Also please be aware of following breaking changes on FreeBSD in next release https://github.com/netbirdio/netbird/pull/2388 https://github.com/netbirdio/netbird/pull/2389
@skillcoder I think the issue is about freebsd being the router, not client routes. Client routes should work and not be much different from darwin
Looks like routing works fine. In both directions. The breaking changes should be no problem for the OPNsense port as it specifies the config file explicitly. I can't say anything about the DNS stuff just that it doesn't find resolv.conf on OPNsense. I'm not a BSD expert so I have no idea if it would be in this location on a standard system. I have disabled setting system DNS in code anyway because in my opinion this feature is to dangerous on a firewall/gateway.
Deployed it on 3 opnsense boxes and configured routes via netbird management. Every firewall rule inside OPNsense is respected and withtout any rule on netbird interface, traffic is blocked (default). So far I can't see any showstoppers in this small test environment. Routing between the OPNsense boxes and access to all networks (with configured firewall and outbound NAT rules) are working as intended.
Hi, just did some tests with Netbird on OPNsense. Thanks @moserpjm for the great work. However I'm not able to route traffic through the Netbird interface. The 100.115.0.0/16 IPs are reachable. But I want to route traffic like @Hobby-Student did (10.x.0.0/16). I installed it with fetch -o /usr/local/etc/pkg/repos/netbird-rc-247.conf https://os-pkg.pjm.co.at/netbird-rc-247.conf
Do I need to compile something on my own? As I understood @Hobby-Student changed something in the initial patch? Where do I find instructions how to do that?
Ideally this could be a switch in the UI to accept routes or not.
Thank you.
