opnsense
Change the default TCP Stack (freebsd) to TCP RACK (RACK-TLP Loss Detection Algorithm for TCP) and support for RFC 8985
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- [x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
Is your feature request related to a problem? Please describe.
FreeBSD 14.3 is still using the default (legacy) tcp stack implementation named "freebsd", but since FreeBSD 14.0 it supports different TCP stacks like "rack" that are more modern and improved. The default tcp stack in freebsd is very sensitive to paketloss, out of order pakets and duplicated acks. I found this out the hard way.
When running opensense as a reverse proxy with caddy i was observing that some of my users had very low speed to me when the traffic was terminating on my reverse proxy. If i NATed the users directly to the webserver(VM) the speed was normal.
A capture via wireshark on the client that was affected showed a very high error rate to any service, but the speed to any conventional speedtest (ookla, fast.com,etc...) was decent even with these absurd error rates:
First I was thinking caddy may be at fault, but running caddy on a ubuntu Server and debian Server showed normal speeds for my testuser.
I then created a fresh freebsd 14.3 install and installed caddy, i was able to reproduce the issue immediately, so I figured that there is some issue in freebsd.
My first conlcusion was that Freebsd is implementing RFC 5681 (a.k.a Slow Start), i tried to get it to work with different parameters, but nothing i tried had a lasting effect for the user (i was able to get the default TCP stack to 20Mbits but not higher)
In the end after struggeling to activate TCP rack at startup, I was able to get it to work and voila, my reverse proxy is now able to serve users with "faulty" connections aswell.
Changing the TCP stack only affects Connections from freebsd and to freebsd (like caddy, GUI etc..) it has no effect on pass-through traffic / firewall traffic.
I would strongly advocate that the opensense project is switching the tcp stack to rack in one of the next major releases
Describe the solution you like
Load TCP rack into the kernel at startup via: Tuneable: tcp_rack_load Value: YES
Enable the new TCP Stack at startup via: Tuneable: net.inet.tcp.functions_default Value: rack
Describe alternatives you considered -Fixing the default tcp stack
-Fixing every ISP out there and eliminating paketloss and re-ordering issues for everyone. Then I punched myself in the nose...
Additional context
Speedtest to my Reverse proxy with the default tcp stack:
Speedtest to my Reverse proxy with TCP Rack as tcp Stack and after rebooting:
Some information about TCP RACK:
https://datatracker.ietf.org/doc/html/rfc8985 https://freebsdfoundation.org/our-work/journal/browser-based-edition/networking-10th-anniversary/rack-and-alternate-tcp-stacks-for-freebsd/ https://freebsd.uw.cz/2025/05/how-to-switch-tcp-stack-and-congestion.html https://freebsdfoundation.org/end-user-stories/netflix-case-study/
I'm not sure what the downsides are for using rack (if any), as this also doesn't influence everyone and the freebsd defaults are the same, we could also opt for an extra choice in the wizard to make the options more explicit.
This is not a bad idea honestly, but there are as well disadvantages. Mainly CPU and Memory, related due to the fact RACK needs to keep a MAP of sent data.
Its discussed more in details in the RFC in section 9. and 10. and as well a vulnerability consideration described in "SACK Attack Detection" https://freebsdfoundation.org/our-work/journal/browser-based-edition/networking-10th-anniversary/rack-and-alternate-tcp-stacks-for-freebsd/
Where to prevent potential missuses it advices to enable TCP_SAD_DETECTION in the kernel.
I'll put it up for discussion here, it certainly doesn't sound like a bad idea, just not fully sure about an automatic default yet.
I'd just add a normal setting somewhere instead of exposing it in the wizard. Since it only affects the local stack its use case is very limited out of the box.
reverse proxying / web application is likely the most common use-case on our end at the moment, changing the default would be relatively easy, adding another toggle somewhere else then tunables is likely less practical as people can't find these easily (and needs modifications on legacy pages we want to replace).
The deployment type page of the wizard isn't that strange in my opinion, as this only affects some deployments.
This affects traffic designated for the system itself (in our case the OPN FW) it will affect/improve all TCP based traffic. Thus any TCP based VPN as well, which can improve stability and performance under certain conditions = user experience.
RACK is not just a tunable e.g "I tuned something", its a wide system change on the TCP STACK. Should not such thing deserve a proper option in the System > Settings, with proper tooltip and documentation?
Should not such thing deserve a proper option in the System > Settings, with proper tooltip and documentation?
That's why I think it's worth adding a setting aside from the wizard in any case, yes.
This affects traffic designated for the system itself
Which in a default factory setup is rather limited. There's DHCP and DNS but WireGuard not, OpenVPN sometimes and somehow IPsec perhaps and all the other things are plugins. :)
I just think it's not a magic bullet and bears risk when somebody fiddles with the stack causing TCP-related panics. It happened before.
I just think it's not a magic bullet and bears risk when somebody fiddles with the stack causing TCP-related panics. It happened before.
True, and as well the benefits will be mainly for connections that suffer by loss e.g Internet circuits or paths that introduce loss into the TCP streams.
Which is mostly webservers (reverse proxies in our case).... openvpn is hardly used in tcp mode, IPsec doesn't use it either. A tunable is likely the best place in my humble opinion (documentation?), when frequently used, the wizard can help pointing people to it.
I also don't think its a magic bullet, adding more settings in legacy pages I would like to prevent for obvious reasons.
I just wanted to chime in again, so after having it running for some days, i can observe a higher interrupt rate, but it is not scary high. I do not think there is a definition for a too high interrupt rate but it has gone up factor 5-10.
Also it has a effect on Unbound when using DoT (DNS over TLS)
Unfortunatly I do not have a before /after of the unbound statistics, but i am quite happy with the statistics
On 15.0 the TCP_SAD_DETECTION and a number of associated sysctls were removed
https://github.com/freebsd/freebsd-src/commit/ea916b64125bf https://github.com/freebsd/freebsd-src/commit/f30ddc76d2a6c
We should probably wait for 15 then to expose this in any way.