Flox-build: Trying to build with sandbox in pure mode and stuck with a cache that's not working.

[stahnma]

Trying to build project: https://github.com/charmbracelet/vhs

Build section of manifest is:

 [build.vhs]
 command = "go mod tidy; go build; mv vhs $out"
 sandbox = "pure"

Then I get an error about a cache unable to write, and a homeless-shelter.

$ flox build
make: Entering directory '/Users/stahnma/go/src/github.com/stahnma/vhs'
/nix/store/yj6m46w8yzc7missw9f4yfmvx1ai9c4q-gnutar-1.35/bin/tar -cf - --no-recursion -T <(/nix/store/45kxi48hp60a6gvammiihx246kspmr7i-git-minimal-2.45.2/bin/git ls-files) > /var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-src.tar
/nix/store/pldbvp0p2z2pa76h17g797vvk201155r-coreutils-9.5/bin/rm -f /var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-buildCache.tar
Rendering vhs build script to /var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-build.bash
Building vhs-0.0.0 in Nix sandbox (pure) mode
set -o pipefail && /nix/store/0gsq84l1imlir3xlqd6s16lgxlbq6fnf-nix-2.18.5/bin/nix --extra-experimental-features "flakes nix-command" build -L --file /nix/store/5yi0l4c512jjd4zbdp4xnsxzcql9g6i5-package-builder-1.0.0/libexec/build-manifest.nix --argstr name "vhs-0.0.0" --argstr srcTarball "/var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-src.tar" --argstr flox-env "/Users/stahnma/go/src/github.com/stahnma/vhs/.flox/run/aarch64-darwin.vhs" --argstr install-prefix "/tmp/store_ea354b05f46f31496bf7cb7755ca888b-vhs-0.0.0"  --argstr buildScript "/var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-build.bash" --argstr buildCache "/var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-buildCache.tar" --out-link "result-vhs" '^*' 2>&1 | tee /var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T/tmp.5KdiT6OsEg-build-vhs.log
this derivation will be built:
  /nix/store/8ra4szix7s825xj3blyhvjf8mhpvmwyg-vhs-0.0.0.drv
building '/nix/store/8ra4szix7s825xj3blyhvjf8mhpvmwyg-vhs-0.0.0.drv'...
vhs> ---
vhs> Input checksums:
vhs> 2f32c91fac0c3e1854a9bce3c678721c  /nix/store/a4z4pn62lcaw087a0nhshd4j0qwar2ih-ea354b05-vhs-src.tar
vhs> 64a6d01c5c49e64bb939eabad5b440f8  /nix/store/jmc9mg3k8msf2ny07x2nzjh2m35kdxwy-ea354b05-vhs-build.bash
vhs> 6eadbb4ec3e269021f9e11a022ccf6da  /nix/store/cadjrjyny945zilg2a4a6gzcin195h27-ea354b05-vhs-buildCache.tar
vhs> ---
vhs> failed to initialize build cache at /homeless-shelter/Library/Caches/go-build: mkdir /homeless-shelter: read-only file system
vhs> failed to initialize build cache at /homeless-shelter/Library/Caches/go-build: mkdir /homeless-shelter: read-only file system
vhs> flox build failed (caching build dir)
ERROR: flox build failed (see /var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T/tmp.5KdiT6OsEg-build-vhs.log)
make: *** [/nix/store/5yi0l4c512jjd4zbdp4xnsxzcql9g6i5-package-builder-1.0.0/libexec/flox-build.mk:307: result-vhs] Error 1
rm /var/folders/sc/rddkr3kd2vg1k1qmy_cx1q900000gn/T//ea354b05-vhs-build.bash
make: Leaving directory '/Users/stahnma/go/src/github.com/stahnma/vhs'
❌ ERROR: Build failed with status: exit status: 2

In what I can only describe as "weird", it works if you run this command 3 times. Not once, Not twice, 3 times. To reproduce this, run rm -rf result-vhs* then you'll need to build this 3 times to get a sandbox enabled build working.

[ghudgins]

potentially: construct a nix post build hook and raise a proper error

[limeytexan]

Mike and I went over this yesterday - there are two things happening here with the "pure" build:

• $HOME set to /homeless-shelter
  • explained to him that users will need to set $HOME=$PWD as the first step of the build in order to get a writable $HOME
  • ... and that we might want to make this something that happens by default, particularly as we preserve $HOME across builds by way of the buildCache
• go is trying to access the network to populate an offline cache
  • that's never going to work anyway
  • I coached Mike through the process of splitting his build into two portions:
    1. one sandbox = "off" build that runs go mod tidy to download stuff
    2. a second sandbox = "pure" build that [essentially] uses the build cache of the first build to perform the second
       • funnily enough, we haven't talked about direct access to ${name-buildCache}, but I don't see why we couldn't make that work

So I think the only takeaways from this issue are:

• set $HOME=$PWD by default in a pure build
• add support for accessing one build's buildCache from another (?)

[stahnma]

Setting $HOME=$PWD kind of helped, but introduced new problems in terms of permissions on the go directory. While it worked once, it left my project folder in a state that I didn't like or want. We're going to need to guides and documents and patterns written to show good ways through builds.

The thing where it worked if you did a thing 3 times is really weird, and felt not all that deterministic.

If we always set HOME to PWD, I'm afraid we're going to encounter some very weird things when using language tools like bundler, npm, pear, cpan, etc. It might be the right path forward, but might also want additional settings. At this point, I would suggest we add to this issue.

Also, if a build is trying to get to a network and unable to, is there a way we can show that as an error and abort?

[zmitchell]

Tasks to be done:

• Set $HOME to $PWD
• Create a ticket to document network access in pure builds (e.g. multi-stage builds)
• Ensure $PWD is writable

@limeytexan couldn't reproduce the 'third time's a charm' issue

[mkenigs]

Tasks to be done:

* Set `$HOME` to $PWD

* Create a ticket to document network access in pure builds (e.g. multi-stage builds)

* Ensure $PWD is writable

@limeytexan couldn't reproduce the 'third time's a charm' issue

@zmitchell can you do that breakdown

[mkenigs]

Closing in favor of https://github.com/flox/flox/issues/2442 and https://github.com/flox/flox/issues/2443