[OS X] Derivation fails with sandbox

[eamsden]

Describe the bug

When running a sandboxed build on OS X, a derivation fails with sandbox-exec: pattern serialization length 71710 exceeds maximum (65535)

If you have a problem with a specific package or NixOS, you probably want to file an issue at https://github.com/NixOS/nixpkgs/issues.

Steps To Reproduce

Create a derivation with a sufficiently large number of inputs, and attempt to build it.

Expected behavior

The derivation builds

nix-env --version output 2.3.7

Additional context

I have done a bit of digging and it seems most likely that this is due to the fact that the OS X sandbox config is created by building a pattern mapping every path in the dependency closure of the derivation to a path in the sandbox individually: https://github.com/NixOS/nix/blob/d761485010cc9d071da17a5d90cf4126a4bf499d/src/libstore/build.cc#L3706-L3729

[siraben]

For me the derivation that fails when I try to build with sandbox on macOS is my home-manager config.

these derivations will be built:
  /nix/store/xfsgnyj7m6pmhjf1qzxawwvsd0nhppif-home-manager-path.drv
  /nix/store/r5s9k5m0apmlk6hn1bysb98pha3aikcm-activation-script.drv
  /nix/store/yclx6cws1qc2gyxn0j1pv70nr802ck69-home-manager-generation.drv
building '/nix/store/xfsgnyj7m6pmhjf1qzxawwvsd0nhppif-home-manager-path.drv'...
sandbox-exec: pattern serialization length 87092 exceeds maximum (65535)
builder for '/nix/store/xfsgnyj7m6pmhjf1qzxawwvsd0nhppif-home-manager-path.drv' failed with exit code 65
cannot build derivation '/nix/store/yclx6cws1qc2gyxn0j1pv70nr802ck69-home-manager-generation.drv': 1 dependencies couldn't be built
error: build of '/nix/store/yclx6cws1qc2gyxn0j1pv70nr802ck69-home-manager-generation.drv' failed

[bbbbbailey]

Bumping because I just experienced this issue with Nix 2.4 (2.4pre20201201_5a6ddb3) when building a home-manager config via nix flakes.

error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
builder for '/nix/store/v75p8mffw7l7pg2pxdwqjfzx89rks8b2-home-manager-path.drv' failed with exit code 65; last 1 log lines:
  sandbox-exec: pattern serialization length 66460 exceeds maximum (65535)
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/8hpb5b7l19y2267dn11cbi4ilx67ribk-home-manager-generation.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/pjjh7a238hwjkvkvhmbmazrzk41jsyz5-user-environment.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/vskz1pxb41sv66irwx2689f39b8hnkld-activation-bbuscarino.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/ak7vwn1z37ymjv192iyl1v2fc5ibkwnx-darwin-system-21.03.20210209.74335e0+darwin4.dca3806.drv' failed to build

[hlolli]

hashtag me2

> nix run .#main-client-build
 warning: unknown setting 'allowUnfree'
warning: Git tree '/Users/hlodversigurdsson/Documents/testapp' is dirty
warning: unknown setting 'allowUnfree'
error: builder for '/nix/store/99swavz43gwjd27w7zx5a2q5gxbrmbxk-node-dependencies-_at_testapp_slash_main-client-1.0.0.drv' failed with exit code 65;
       last 1 log lines:
       > sandbox-exec: pattern serialization length 89274 exceeds maximum (65535)
       For full logs, run 'nix log /nix/store/99swavz43gwjd27w7zx5a2q5gxbrmbxk-node-dependencies-_at_testapp_slash_main-client-1.0.0.drv'.
error: 1 dependencies of derivation '/nix/store/mgi4b6a83rxqmzkv0603mia72fccwj1c-node-shell-_at_testapp_slash_main-client-1.0.0.drv' failed to build
> nix log /nix/store/99swavz43gwjd27w7zx5a2q5gxbrmbxk-node-dependencies-_at_testapp_slash_main-client-1.0.0.drv
warning: unknown setting 'allowUnfree'
sandbox-exec: pattern serialization length 89274 exceeds maximum (65535)
> nix --version
warning: unknown setting 'allowUnfree'
nix (Nix) 2.4pre20210503_6d2553a

[hlolli]

So it seems the generated sandbox profile has an s-expression exceeding the limit

when running with nix --debug

executing builder '/nix/store/ldfws5lqyz0irmb3bcvnjawyhsva38xq-bash-4.4-p23/bin/bash'
sandbox setup: Generated sandbox profile:
sandbox setup: (version 1)
sandbox setup: (deny default (with no-log))
sandbox setup: (import "sandbox-defaults.sb")
sandbox setup: (allow file-read* file-write* process-exec
sandbox setup:      (subpath "/nix/store/8xywf3whgq08045glmv99v2bgxmhvh00-node-dependencies-_at_testapp_slash_main-client-1.0.0")
sandbox setup: )
sandbox setup: (allow file-read* file-write* process-exec
sandbox setup:        (subpath "/System/Library/Frameworks")
sandbox setup:  (subpath "/System/Library/PrivateFrameworks")
sandbox setup:   (literal "/bin/bash")
sandbox setup:   (literal "/bin/sh")
sandbox setup:     (literal "/dev/random")
sandbox setup:         (literal "/dev/urandom")
sandbox setup:        (literal "/dev/zero")
sandbox setup:   (literal "/nix/store/006jl3mw0xiqwrfnmm0dn6zxkpaqkdar-http2-wrapper-1.0.3.tgz")
sandbox setup:         (literal "/nix/store/011f5148hdsm6yznj4rl30j4jhzlhrb6-is-buffer-1.1.6.tgz")
sandbox setup:     (literal "/nix/store/026dnivlzqrsdrf8pkc47hh5f3arm9kk-final-form-4.20.2.tgz")
....

I'm assuming splitting these statements into smaller blocks will fix this.

[domenkozar]

@toonn something to put on the list for https://opencollective.com/nix-macos

[domenkozar]

@toonn I went through Nix issues concerning macOS sandboxing and this is the only one left reported to fix.

It needs research when the limit kicks in and what's the workaround (maybe we have to use import and split the expression if it becomes too long).

[hlolli]

@toonn I went through Nix issues concerning macOS sandboxing and this is the only one left reported to fix.

It needs research when the limit kicks in and what's the workaround (maybe we have to use import and split the expression if it becomes too long).

I'd not be surprised that the limit is 65535, but that's hard to determine from counting the packages, because I think there's one sandbox item entry for every nix store path. I did read a bit about the sandbox specification from osx developer manual, and started attempting to write a fix by batching these in multiple files (which is totally possible, importing .sb files). But my lack of c++ knowledge, and especially modern c++ as is in the nix repo, got the better of me.

sandbox-exec: pattern serialization length 89274 exceeds maximum (65535)

[stale[bot]]

I marked this as stale due to inactivity. → More info

[thefloweringash]

Having a sandbox on macOS is still a worthy goal.

and started attempting to write a fix by batching these in multiple files (which is totally possible, importing .sb files).

For the record, were you able to confirm that importing from multiple files is sufficient to bypass the limit? I tried experimenting a little, and I think the serialization limit is at a lower level.

I made two files with ~800 store paths in them, and a top-level file to import them.

• If I include either file by itself, sandbox-exec works.
• If I include both, the computed size is 97830 and this produces the pattern serialization length 97830 exceeds maximum (65535) error.
• If I combine the files, the error message is exactly the same, down to the same computed size.
• If I include all three files (slice 1, slice 2, combined), the error message is still the same. Which to me suggests some kind of unique representation for each path, and a format not related to the input.

If it's not possible to encode the strict nix sandboxing scheme, could we still have a sandbox that prevents access to things outside of the store path? I think this could still help a lot with purity.

[lilyball]

I just ran into this myself.

What I find especially interesting/frustrating is that I ran into it while performing a nix flake check after enabling additional services on one of my outputs.nixosConfigurations. That output isn't for the current platform, so I've configured Nix with a remote builder for aarch64-linux. It looks like it's failing on a trivial wrapper /nix/store/bbblnlsl5hwxbv4s145cf7vh8wyxqnqp-deploy-rs-check-activate.drv which is an aarch64-darwin derivation that just runs a simple shell script over an aarch64-linux derivation /nix/store/x499w8xz2zrjniz1lb5y3cmrnm3zi8si-activatable-nixos-system-pyra-22.05.20220428.6766fb6.drv. So it's generating a massive sandbox profile with 1213 literal/subpath entries all so it can just check for the presence of 2 files.

I really want to keep the macOS sandbox. If there's no way to get around this by splitting things up, I wonder if we could do something like generate regexes that match multiple paths with a single pattern. I don't know what regex dialect it is but I've never seen one that doesn't have alternations. At the very we could remove /nix/store duplication by merging multiple paths together. We could also truncate each (subpath "/nix/store/…") entry after the store hash, such as (regex #"^/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-.*") instead of (subpath "/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-iana-etc-20211124"), as that would save some characters that are presumably contributing to the serialization size as well.

Assuming this works, there is a limit to how much space could be saved this way (and I question whether constructing a massive regex with alternations would affect performance; if it internally converts literals and subpaths into regexes already then there should be no difference), but anything to increase the size would help. And if we can accurately predict the serialization size we could also have a degraded fallback mode where we start allowing extra files if it means writing fewer rules in order to stay under the size. Jumping all the way to (subpath "/nix/store") if we can't reasonably stay in the limit is still significantly better than abandoning the sandbox.

[lilyball]

Incidentally it sounds like older versions of macOS had a command sandbox-simplify, it doesn't exist anymore but if anyone has a sufficiently old version of macOS that still has it I'm wondering if it does anything to a large Nix-generated sandbox profile.

[2xsaiko]

Just hit this as well on a pkgs.linkFarm with ~3000 entries:

nix build 'git+https://git.dblsaiko.net/nix-mc?rev=3e53544c6bb8758d3ff5357c7ba6dee616ad1ac4#minecraftPackages.aarch64-darwin.1_19_1.assets'

[carlthome]

Getting this with Home Manager currently over in http://github.com/carlthome/dotfiles - is there some quick workaround that does not involve disabling the sandbox?

[samuel-gelineau-at-well-dot]

The following workaround has worked for me. The error message mentions a .drv file. If I run the following command on that drv file:

nix-store --option sandbox false --use-output --query --force-realise --requisites --include-outputs /nix/store/[...].drv

then the next time I try to run the command which was failing with "pattern serialization length exceeded", that command now succeeds.

The key bit in the above command is --option sandbox false. So I suspect you can also fix the problem by disabling the sandbox in other ways, such as by setting sandbox = false in your nix configuration or by running a simpler nix command involving --option sandbox false and the .drv file. But the above is what has worked for me today, so I wanted to share.

[jakubgs]

Is there a way to apply __noChroot = true dynamically to all builds of certain kind?

I know I can set sandbox = false in nix.conf, but that would result in us not using sandbox for Android builds, and I would rather keep that. What I would love is to do __noChroot = isDarwin, but for all packages. Especailly yarn2nix-moretea, since it can often result in errors like this due to size of the derivation that includes all node_modules for a big project.

I was looking at yarn2nix derivation and I'm not sure it's possible to apply __noChroot = isDarwin by using any overrides.

[n8henrie]

Just ran into this while building my nix-darwin config, with the only change being adding myself to trusted-users or extra-trusted-users. Oddly if I add only myself to trusted-users it succeeds, but if I keep root there as well, it fails.

[lilyball]

I recently hit this with nix-darwin and ended up having to add the following to my config:

    system.systemBuilderArgs = mkIf (config.nix.settings.sandbox == "relaxed") {
      sandboxProfile = ''
        (allow file-read* file-write* process-exec mach-lookup (subpath "${builtins.storeDir}"))
      '';
    };

[n8henrie]

Thanks for the tip @lilyball -- is system.systemBuilderArgs documented anywhere? I assume it is not showing up at search.nixos.org/options (nor in the manual) because internal = true;?

[Pegasust]

Thanks for the tip @lilyball -- is system.systemBuilderArgs documented anywhere? I assume it is not showing up at search.nixos.org/options (nor in the manual) because internal = true;?

It is indeed because internal=true. See the backend of nixos-search here.

https://github.com/NixOS/nixos-search/blob/00f61f52aefe60b18059ec13201d9c94bdc04045/flake-info/assets/commands/flake_info.nix#L105

For nix-darwin, simple code search should push us started somewhere. From the looks of it, nix-darwin just pushes it up and probably uses nixos' builder somehow.

https://github.com/search?q=repo%3ALnL7%2Fnix-darwin%20systembuilderargs&type=code

https://github.com/LnL7/nix-darwin/blob/e236a1e598a9a59265897948ac9874c364b9555f/modules/system/default.nix#L55-L62

For nixos, there are some matches to get us started

https://github.com/search?q=repo%3ANixOS%2Fnixpkgs%20systemBuilderArgs&type=code

[n8henrie]

@Pegasust indeed, I searched nix-Darwin and had linked to the nixpkgs source in my comment. Thanks for the tip about the options search filter!

[n8henrie]

Hmmm, @lilyball's system.systemBuilderArgs workaround doesn't seem to be working for me.

I have that snippet and I have nix.settings.sandbox = "relaxed";, but still hitting this error.

[lilyball]

@n8henrie What's the derivation that it's failing on for you? The workaround works for me because it's the top-level system builder that had the overly large sandbox profile, but for you perhaps it's another derivation that causes the problem.

[n8henrie]

Looks like it's ultimately my home-manager config (failing at my system config, but the traceback seems to lead back to HM).

[amarshall]

For Home Manager, see https://github.com/nix-community/home-manager/pull/3729

[n8henrie]

Interesting, thanks. Fixed link: https://github.com/nix-community/home-manager/pull/3729

I had tried adding this overlay to my flake to no avail:

      (_: prev: {
        home-manager = prev.home-manager.overrideAttrs {
          sandboxProfile = ''
            (allow file-read* file-write* process-exec mach-lookup (subpath "${builtins.storeDir}"))
          '';
        };
      })

With the __noChroot workaround I get an error about sandbox = true;, even though mine is set to "relaxed", I must have something configured wrong.

EDIT:

$ nix eval .#darwinConfigurations.NatePro.config.nix.settings.sandbox
false

EDIT2: Looks like I probably need the activationScripts and such as well, I just tried with the overlay.

[n8henrie]

If anyone is looking to reproduce the error manually, there is likely a better way, but here are my steps:

1. darwin-rebuild --flake . switch gives me:

error: builder for '/nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv' failed with exit code 65;                                                                                         │
│       last 1 log lines:                                                                                                                                                                                                                    │
│       > sandbox-exec: pattern serialization length 75123 exceeds maximum (65535)                                                                                                                                                           │
│       For full logs, run 'nix log /nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv'.                  

2. nix build --debug that derivation:

$ nix build --debug \
    /nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv \
    &> sandbox.sb

3. Manually edit sandbox.sb to strip out the scheme code, starting with (version 1)
4. Using src/libstore/build/local-derivation-goal.cc as an example:

$ sandbox-exec -f ./sandbox.sb -D _GLOBAL_TMP_DIR=/tmp/foo echo fake                
sandbox-exec: pattern serialization length 75123 exceeds maximum (65535)                                                                                                                        

(Would likely replace echo fake with nix build the_derivation.drv once the sandbox error is fixed).

[n8henrie]

Trying @lilyball's suggestion:

We could also truncate each (subpath "/nix/store/…") entry after the store hash, such as (regex #"^/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-.*") instead of (subpath "/nix/store/zs08rlx67p6vjjbk4y3c75ylr9r56xms-iana-etc-20211124"), as that would save some characters that are presumably contributing to the serialization size as well.

I used the following nvim replacement:

:%s:subpath "\(/nix/store/[^-]*\)-.*":regex #"\1-.*": 

Running the below line count pre and post gives the same result, so it doesn't look like any additional paths can be removed (I thought perhaps there would be multiple binaries from the same package listed):

$ sed -e 's/^[[:blank:]]*//' -e 's/[[:blank:]]*$//' < sandbox.sb  | sort -u | wc -l                                                                                                                               
1268

Afterwards, I get farther(?), so it seems like this is probably a valid temporizing measure:

$ sandbox-exec \
    -f ./sandbox.sb \
    -D _GLOBAL_TMP_DIR=/tmp/foo \
    $(type -p nix) build /nix/store/8c1h3pifzs3xpfazfvcqbikgd5mf61j9-darwin-system-23.05.20230908.3fbd2a2+darwin4.511177f.drv
sandbox-exec: execvp() of '/run/current-system/sw/bin/nix' failed: Operation not permitted      

Rereading the debug output, looks like I'm using the wrong approach for building, but this doesn't work either (even though I have execute permissions for this bash):

$ sandbox-exec \
    -f ./sandbox.sb \
    -D _GLOBAL_TMP_DIR=/tmp/foo \
    /nix/store/7avcas47b0g25c9bk7m4xb7p5p636qs7-bash-5.2-p15/bin/bash \
    -e /nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh
sandbox-exec: execvp() of '/nix/store/7avcas47b0g25c9bk7m4xb7p5p636qs7-bash-5.2-p15/bin/bash' failed: Operation not permitted

[jsoo1]

Is it possible that this could be fixed with one more layer of indirection?

Suppose we were to create a chroot and mount in all the store dependencies, could we reduce the number of store paths in the sandbox profile to only the chroot directory?

Edit: I suppose the lack of bind mounts would make this impossible.

[lilyball]

I do wonder how much mileage we could get by doing things like having a single regex that matches multiple store paths (as an extension on the idea of "use a regex to match up to the hash") to save even more characters, but I don't know how the "pattern serialization length" is calculated and it's plausible that it's already basically joining all paths with the same perms into a single pattern internally.

[jsoo1]

I found a gist that had some notes on experiments done for this problem which seemed to suggest the patterns were all reduced prior to sending to the kernel. Let me see if I can dig it up...

https://gist.github.com/rrbutani/16c7c66202ed47b2a5dd8cb5e916fb0a

Their experiments seemed to suggest that sandbox-simplify was subsumed by sandbox-exec.

I do wonder what might be done with sandbox.h after some perusal. Though I have the hardest time digging up apple's headers (if indeed it is public).