nix-community
network sysctls
G'day numtide,
Thanks for all the great numtide projects! I've been learning Nix and you guys are definitely leaders. Thank you.
In the spirit of giving back, I was reading the blog about SrvOS ( https://numtide.com/blog/donating-srvos-to-nix-community/ ), and took a quick look. I noticed some standard TCP performance tweaks I apply are missing (most importantly the TCP buffer sizes), and these tweaks will likely be suitable in the vast majority of cases. Therefore, I submit this little pull request for your thoughts.
It's interesting to see that you ARE changing to BBR-TCP by default. This is probably mostly safe across the WAN, but BBR is not a silver bullet, and in many cases (like any connection with low RTTs) is likely to be making performance worse. I did a lot of testing at Edgecast CDN, and we determined NOT to change to BBR, but would selectively switch to BBR for some destination subnets
https://edg.io/technical-articles/improving-network-performance-with-dynamic-congestion-control/
Thanks again, Dave
Thanks for the slides.
Even BBR2 isn't merged into the kernel, so currently BBR1 is the only available. The team working on BBR2 and L4S/Prague ( https://www.rfc-editor.org/rfc/rfc9330.html ) do have a branch with bbr2 in it. I guess something that might be cool is to get Nix to apply the BBR2 patches. https://github.com/L4STeam/linux/blob/56eae305cddf172b87c54d8a61db8d1e9e2204f0/net/ipv4/tcp_bbr2.c#L1304
Apparently BBR3 isn't in the repo :( https://github.com/search?q=repo%3AL4STeam%2Flinux+bbr3&type=code
On Fri, Nov 8, 2024 at 10:10 PM Jörg Thalheim @.***> wrote:
@.**** commented on this pull request.
In nixos/server/default.nix https://github.com/nix-community/srvos/pull/556#discussion_r1835286817:
@@ -111,10 +111,56 @@ ''; };
use TCP BBR has significantly increased throughput and reduced latency for connections
https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html
In some cases, TCP BBR can significantly increase throughput and reduce latency,
however this is not true in all cases, and should be used with caution
Ok. So my assumption here, when I enabled bbr here, was that it would help for the typical server (1Gpbs-10Gpbs uplink, running in some data-center with reasonable global peering). That's why we also only put it in the "server" profile while keeping the default for the "desktop" profile.
But your data suggest that it would be actually worse for these cases. Interestingly your 4% performance improvement seem to match the result that Youtube reported as well.
Which version of BBR did you use for your testing. It looks like since 2023 we now also have BBRv3, with some enhancements: https://datatracker.ietf.org/meeting/117/materials/slides-117-ccwg-bbrv3-algorithm-bug-fixes-and-public-internet-deployment-00
— Reply to this email directly, view it on GitHub https://github.com/nix-community/srvos/pull/556#discussion_r1835286817, or unsubscribe https://github.com/notifications/unsubscribe-auth/APMCHTRATJ6DBQQG2OENDALZ7WRNBAVCNFSM6AAAAABRFDAREKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDIMRVGIZTAMJVGY . You are receiving this because you authored the thread.Message ID: @.***>
-- Regards, Dave Seddon +1 415 857 5102
@randomizedcoder if your recommendation would be to disable bbr for most server usage, maybe we should drop it from srvos than.
