NixOS
Add sandboxed building for FreeBSD using jails
Motivation
Build isolation is good! In Linux, this is accomplished with namespaces (containers). The equivalent technology on FreeBSD is jails.
Context
This is part of my ongoing project to make FreeBSD a first class citizen in the nix world.
This was a fairly simple patch, just needed to add parallel implementations for all the sandboxed Linux build pieces.
The most fragile part of this implementation is the fact that there is a lot of global state that gets set up in order to construct the jail - the chroot dir in the nix store, the nullfs mounts (the FreeBSD equivalent of a bind mount), and the jail ID itself. Lots of steps have been taken to make sure these all get cleaned up, both at the end of the build and at the start of any rebuilds. It seems to be resilient to interruption.
This has been live-fire tested with my fork of nixpkgs for FreeBSD. It is able to build the stdenv without issue.
Please squash-merge this PR! It includes some changes that were later reverted, which don’t belong in this repository but instead in the FreeBSD ports repository.
Priorities and Process
Add :+1: to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.
This is very cool! But before supporting any new sandboxing methods, I would ideally like to de-spagetti the code that we already have. Are you up for helping out with that too?
I am not entirely sure, but to start, see in https://github.com/NixOS/nix/pull/8901 how I have some platform specific .cc and .hh files.
More broadly it's a process of untangling the spaghetti where we cannot tell what exactly we're gonna get until we do it :)
Ideally we would have subclasses of LocalDerivationGoal that provide the platform-specific code for building (such as sandboxing). Or create some abstract interface for build execution that LocalDerivationGoal can use.
Or create some abstract interface for build execution that
LocalDerivationGoalcan use.
Yes I would prefer that. I made LocalDerivationGoal a DerivationGoal subclass because it was an easy first step to split things up, but I don't think deep inheritance hierarchies is how this stuff aught to work.
I gave it a shot. Unfortunately, due to my lack of deep understanding of the requisite linux and macos internals, I wasn't able to do a whole lot more than moving verbose chunks of ifdef'd code into platform-specific files. However, I think this makes it a lot more readable.
Discussed in the Nix team meeting. Conclusions:
- Assigned to @ericson2314 to follow-up
- Can land the cleanups in a preperatory PR
details:
-
@edolstra: The refactoring is welcome, but we don't have the capacity to maintain the FreeBSD code
- @roberth: (maintain it ourselves)
-
@rhelmot also revived the FreeBSD stdenv in Nixpkgs (https://github.com/NixOS/nixpkgs/pull/254801)
-
@roberth: Would be great if we could make them code owner or similar
-
@edolstra: Mac bugs are painful because foreign to most maintainers and block features
- @roberth: FreeBSD might be easier than MacOS for that (and also uncover the same bugs, making MacOS less of a trouble)
-
Proposal (@thufschmitt): If the refactorings allow having the FreeBSD sandbox non-intrusive, then we can accept it without commitments on maintaining it ourselves (like we already do for the broader FreeBSD support).
- Agreement
-
@ericson2314: In particular, we can merge the preperatory cleanups before we've decided on the FreeBSD-specific code.
Assigned to @ericson2314 to follow-up
Extra note from me: hope jumping through these hoops is not too disheartening @rhelmot. You've done very good work here and in Nixpkgs on this stuff! I look forward to working with you on getting it all merged.
I'm fine with being made code owner, and I'm fine with rebasing my changes on top of a prep PR which carves out spaces for the sandbox to live. The categories I was able to pull out in terms of my attempted refactor were very much best-effort descriptions of what was already there.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-02-12-nix-team-meeting-minutes-123/39775/1
@Ericson2314 @edolstra I've made some clean-ups locally for supporting WASM derivations in my local code. The clean-up involves removing LocalDerivationGoal and refactoring the building into a separate class that DerivationGoal uses. I currently have HookBuilder, NativeBuilder, and WasmBuilder (which isn't relevant to this PR but is what my goal is). You could reasonably split out NativeBuilder into LinuxBuilder, BSDBuilder, etc.
EDIT:
This is the interface I made between DerivationGoal and the ways of building:
https://github.com/L-as/nix/blob/ce6d800bb513728b1ff92916ca5f64929eb698fb/src/libstore/build/builder-interface.hh (wasm-derivations branch)
@rhelmot I (sleepily) talked to @L-as a bit about this at Nix Con NA, but i am hoping that you and him and I and work together on these cleanups, and between the 3 of us it will be less onerous :).
https://github.com/NixOS/nix/pull/13276 Now that this is done, I think my "cleanup the current big mess before new features" objective is thoroughly met!
BTW https://github.com/Ericson2314/nix/tree/freebsd-older is, for archival purposes. A slightly cleaned up history that ends up in the exact state this PR was in before I start pushing to it.
OK this is now almost rebased. And the first commit is in fact merged thanks to https://github.com/NixOS/nix/pull/13281 already.
What is left is rebasing this over https://github.com/NixOS/nix/pull/13276, which will have big conflicts, but ought to lead to a really nice result.
OK! I haven't tested that it works on FreeBSD, but it does cross build. The rebase, while heavily manual, we less bad than I thought.
Current status, I was able to install using the single-user install and than I create manually the users for the multi-user daemon. However currently the sandbox fails to start and overrides /etc, so not a recommendation to test this further if this is not fixed unless you are in a VM.
root@nixbsd:~ # cat default.nix
derivation {
name = "hello2";
system = "x86_64-freebsd";
builder = "/bin/sh";
args = [ "-c" "echo hello world > $out"];
}
root@nixbsd:~ # /home/nix-test/.nix-profile/bin/nix-build --sandbox --substituters ''
this derivation will be built:
/nix/store/9qzay9z0g7v26vbpvwwf49867dwrv4pq-hello2.drv
error: Failed to create jail (isolated network): No such file or directory
root@nixbsd:~ # cat /etc/passwd
root::0:0:Nix build user:/build:/noshell
nixbld::30001:30000:Nix build user:/build:/noshell
nobody:*:65534:65534:Nobody:/:/noshell
I found the corresponding Lix patch (not yet merged) https://gerrit.lix.systems/c/lix/+/1663?usp=search and updated some things, but it is not yet fully in sync. It would be good if @rhelmot and @artemist could finish this one off.
rebased. not sure if I'll have time to clean up the review comments anytime soon
The rebase seemed to have lost several pieces of the original PR which I had to add back in. It also looks like some pieces from the lix version of this PR got spliced in but they actually duplicated the ifconfig shell call... so I guess the review comments are addressed now?
I will try to do more std::filesystem::path myself or with @vinayakankugoyal — the LLM can crank through this sort of thing fairly well.
Let's get another review from @Mic92 on the logic itself, but then I think it's a wrap!
I'm happy to work with @xokdvium to land these C++ best practices changes.
@rhelmot what would be a better use of your time and unique knowledge is retroactively reviewing the rest of what I did in https://github.com/NixOS/nix/pull/13281 that this PR doesn't touch. In particular it would be nice if we can get rid of the FreeBSD-only argument to _deletePath.