rules_nixpkgs
rules_nixpkgs copied to clipboard
tweag
go_tool_binary fails under remote execution (mktemp: command not found)
Describe the bug
When using remote execution as described in the tutorial, go builds fail with the following message:
INFO: Remote execution message for GoToolchainBinaryBuild @go_sdk//:builder: Action details (uncached result): http://bb-browser.example.com:80/blobs/sha256/historical_execute_response/8b82519fa711e6ec14f57fb9175fe617bfff0019e4233d866b55292c4964ed67-925/
ERROR: /home/malte/.cache/bazel/_bazel_malte/00fb985034c463ac103394fddf3349bf/external/go_sdk/BUILD:27:15: GoToolchainBinaryBuild external/go_sdk/builder [for tool] failed: (Exit 1): bash failed: error executing command (from target @go_sdk//:builder)
(cd /home/malte/.cache/bazel/_bazel_malte/00fb985034c463ac103394fddf3349bf/execroot/constellation && \
exec env - \
/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash -c 'GOMAXPROCS=1 GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/k8-opt-exec-C7777A24-ST-894cc94080c3/bin/external/go_sdk/builder -trimpath -ldflags '\'''\'' external/io_bazel_rules_go/go/tools/builders/ar.go external/io_bazel_rules_go/go/tools/builders/asm.go external/io_bazel_rules_go/go/tools/builders/builder.go external/io_bazel_rules_go/go/tools/builders/cgo2.go external/io_bazel_rules_go/go/tools/builders/compilepkg.go external/io_bazel_rules_go/go/tools/builders/cover.go external/io_bazel_rules_go/go/tools/builders/edit.go external/io_bazel_rules_go/go/tools/builders/embedcfg.go external/io_bazel_rules_go/go/tools/builders/env.go external/io_bazel_rules_go/go/tools/builders/filter.go external/io_bazel_rules_go/go/tools/builders/filter_buildid.go external/io_bazel_rules_go/go/tools/builders/flags.go external/io_bazel_rules_go/go/tools/builders/generate_nogo_main.go external/io_bazel_rules_go/go/tools/builders/generate_test_main.go external/io_bazel_rules_go/go/tools/builders/importcfg.go external/io_bazel_rules_go/go/tools/builders/link.go external/io_bazel_rules_go/go/tools/builders/pack.go external/io_bazel_rules_go/go/tools/builders/read.go external/io_bazel_rules_go/go/tools/builders/replicate.go external/io_bazel_rules_go/go/tools/builders/stdlib.go external/io_bazel_rules_go/go/tools/builders/stdliblist.go external/io_bazel_rules_go/go/tools/builders/path.go')
# Configuration: a398af2ac26d49408999f021c57935c95bc395a73b4a2060b75e3c4ccf4292dc
# Execution platform: //:ubuntu-act-22-04-platform
/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash: line 1: mktemp: command not found
/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash: line 1: mktemp: command not found
warning: GOPATH set to GOROOT () has no effect
build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined
This is a similar (working) action when using Ubuntu and vanilla Bazel locally:
# Configuration: 7ef5b98b65eac4fc464889e87ad35f3b0e52ab8f56c97ba8e1878ec35a0c80e6
# Execution platform: //:ubuntu-act-22-04-platform
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 65fcd5f445b9bbab7d9041ac9c2abcf79c07f40a07e5e5308062e3224a7b10a8, execution platform: //:ubuntu-act-22-04-platform]
(cd /root/.cache/bazel/_bazel_root/00fb985034c463ac103394fddf3349bf/execroot/constellation && \
exec env - \
/bin/bash -c 'GOMAXPROCS=1 GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/k8-opt-exec-3EDBA836-ST-724252c57a52/bin/external/go_sdk/builder -trimpath -ldflags '\'''\'' external/io_bazel_rules_go/go/tools/builders/ar.go external/io_bazel_rules_go/go/tools/builders/asm.go external/io_bazel_rules_go/go/tools/builders/builder.go external/io_bazel_rules_go/go/tools/builders/cgo2.go external/io_bazel_rules_go/go/tools/builders/compilepkg.go external/io_bazel_rules_go/go/tools/builders/cover.go external/io_bazel_rules_go/go/tools/builders/edit.go external/io_bazel_rules_go/go/tools/builders/embedcfg.go external/io_bazel_rules_go/go/tools/builders/env.go external/io_bazel_rules_go/go/tools/builders/filter.go external/io_bazel_rules_go/go/tools/builders/filter_buildid.go external/io_bazel_rules_go/go/tools/builders/flags.go external/io_bazel_rules_go/go/tools/builders/generate_nogo_main.go external/io_bazel_rules_go/go/tools/builders/generate_test_main.go external/io_bazel_rules_go/go/tools/builders/importcfg.go external/io_bazel_rules_go/go/tools/builders/link.go external/io_bazel_rules_go/go/tools/builders/pack.go external/io_bazel_rules_go/go/tools/builders/read.go external/io_bazel_rules_go/go/tools/builders/replicate.go external/io_bazel_rules_go/go/tools/builders/stdlib.go external/io_bazel_rules_go/go/tools/builders/stdliblist.go external/io_bazel_rules_go/go/tools/builders/path.go')
This seems to be related to the following code in rules_go:
https://github.com/bazelbuild/rules_go/blob/4211c6d32ee475a8fde1cfc91571e7c0bed67af4/go/private/rules/binary.bzl#L462
To Reproduce
- Follow the tutorial
- Use the nixpkgs go toolchain
- Build any go binary
Expected behavior
The go binary should build successfully
Environment
- Local: NixOS 23.04 (amd64), Remote: buildbarn with ubuntu 22.04 act
- Version of the code: latest head (3e21df2ee2d5495949b6777a841b22d158b4399d)
Additional context
Maybe we can install mktemp via nixpkgs?
I think I fully understand the problem now and can offer a solution.
This behavior only occurs when the local Bazel is coming from nixpkgs. Bazel from nixpkgs has a set of patches that slightly change the behavior compared to vanilla Bazel.
More specifically, there is this issue: https://github.com/NixOS/nixpkgs/issues/94222
And this set of patches that are relevant:
-
override hardcoded path to
/bin/bashto point to a bash from nixpkgs -
default initialize
$PATHto a bash and a minimal set of tools (including coreutils which contains mktemp) -
the actual
$PATHvariable used as a fallback. On my build of Bazel, this containsbashandmktempwith the following store paths:-
/nix/store/y9gr7abwxvzcpg5g73vhnx1fpssr5frr-coreutils-9.3/bin/mktemp -
/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash
-
What happens?
When using this patched Bazel locally with BuildBarn runners remotely, the following happens (to the best of my understanding):
Analysis phase
During the analysis phase, the patched Bazel evaluates this rule and sets the shell to /nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash ^1.
(When using vanilla Bazel, the shell is instead set to /bin/bash).
https://github.com/bazelbuild/rules_go/blob/4211c6d32ee475a8fde1cfc91571e7c0bed67af4/go/private/rules/binary.bzl#L462-L473
Execution phase (local)
When performing a local build, both store paths (bash and mktemp) are in the local nix store.
run_shell uses the hardcoded bash from the nix store. While the whole env including $PATH would be empty under vanilla Bazel, this patch adds coreutils from the nix store to the path containing from the coreutils nix store path and the action can run successfully.
Execution phase (RBE)
During a remote build, run_shell still uses the hardcoded bash from the nix store. However, the env actually empty this time, since this patch is only applicable to local execution. This means the $PATH variable is empty and the bash from nixpkgs sets the $PATH to /no-such-path, leading to mktemp: command not found.
How to fix this?
Short term, I found that I can set --shell_executable=/bin/bash to ensure ctx.actions.run_shell picks a working shell on the runner. Long term, I think we should make the bash from nixpgks work using one of the approaches below.
We could change this patch to point to a bash that includes coreutils in it's embedded $PATH or get this patch to work on the remote runner.
In any case, we also need to upload bazel with its dependencies to the NFS server.
Thank you for raising this issue! I haven't gone through all the details, yet, but a Go related mktemp failure also occurred here: https://github.com/tweag/rules_nixpkgs/pull/423#issuecomment-1743356411 . I've worked around it in https://github.com/tweag/rules_nixpkgs/pull/423/commits/925f8e45901af3f00157fde62a2e28adfa3a89ed and opened https://github.com/NixOS/nixpkgs/pull/258608 for an upstream fix.
Thank you for taking a look. While both issues are related to Go and mktemp, they have different root causes.