`nix-build --rebuild` does not work with remote builders

[Tristano8]

Describe the bug

nix build doesn't use a remote builder, even when passed explicitly via --builder, unless explicitly telling it to ignore local builds. It seems like the nix scheduler thinks that it can build an x86_64-linux derivation locally, then at build time discovers that it cannot actually do that

Steps To Reproduce

1. On a silicon Mac, set up a remote builder in /etc/nix/nix.conf for x86_64-linux

eg:

builders = ssh://mybuilder x86_64-linux - 6
builders-use-substitutes = true

3. Run nix build -Lv --rebuild nixpkgs#legacyPackages.x86_64-linux.hello
4. See the following error:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/amplsdcav0g3qp9srkkfnqlkr8zc3sn8-hello-2.12.1.drv', but I am a 'x86_64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

5. Repeat step 3 with --max-jobs 0
6. Observe no error

Expected behavior

The result from step 6 should occur without passing --max-jobs

nix-env --version output nix-env (Nix) 2.18.1

Additional context

This occurs on multiple silicon machines (M1 and M3) and across multiple nix versions (2.21.1 and 2.18.1 at least).

Full error logs:

error-log.txt

My nix show-config:

accept-flake-config = false
access-tokens = <REDACTED>
allow-dirty = true
allow-import-from-derivation = true
allow-symlinked-store = false
allow-unsafe-native-code-during-evaluation = false
allowed-impure-host-deps = /System/Library /bin/sh /dev /usr/lib
allowed-uris = 
allowed-users = *
auto-allocate-uids = false
auto-optimise-store = false
bash-prompt = 
bash-prompt-prefix = 
bash-prompt-suffix = 
build-hook = /nix/store/6zi5fqzn9n17wrk8r41rhdw4j7jqqsi3-nix-2.18.1/bin/nix __build-remote
build-poll-interval = 5
build-users-group = nixbld
builders = ssh://remote-builder x86_64-linux - 6
builders-use-substitutes = true
commit-lockfile-summary = 
compress-build-log = true
connect-timeout = 0
cores = 12
darwin-log-sandbox-violations = false
diff-hook = 
download-attempts = 5
download-speed = 0
eval-cache = true
experimental-features = flakes nix-command
extra-platforms = aarch64-darwin
fallback = false
flake-registry = https://channels.nixos.org/flake-registry.json
fsync-metadata = true
gc-reserved-space = 8388608
hashed-mirrors = 
http-connections = 25
http2 = true
id-count = 128
ignore-try = false
impersonate-linux-26 = false
keep-build-log = true
keep-derivations = true
keep-env-derivations = false
keep-failed = false
keep-going = false
keep-outputs = false
log-lines = 10
max-build-log-size = 0
max-free = 2147483648
max-jobs = 1
max-silent-time = 0
max-substitution-jobs = 16
min-free = 1073741824
min-free-check-interval = 5
nar-buffer-size = 33554432
narinfo-cache-negative-ttl = 3600
narinfo-cache-positive-ttl = 2592000
netrc-file = /etc/nix/netrc
nix-path = /Users/hughdavidson/.nix-defexpr/channels nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixpkgs /nix/var/nix/profiles/per-user/root/channels
plugin-files = 
post-build-hook = 
pre-build-hook = 
preallocate-contents = false
print-missing = true
pure-eval = true
require-drop-supplementary-groups = false
require-sigs = true
restrict-eval = false
run-diff-hook = false
sandbox = false
sandbox-fallback = true
sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /bin/bash /bin/sh /private/tmp /private/var/tmp /usr/lib
secret-key-files = 
show-trace = false
ssl-cert-file = /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt
stalled-download-timeout = 300
start-id = 56930
store = auto
substitute = true
substituters = <REDACTED>
sync-before-registering = false
system = x86_64-darwin
system-features = benchmark big-parallel nixos-test
tarball-ttl = 3600
timeout = 0
trace-function-calls = false
trace-verbose = false
trusted-public-keys = <REDACTED>
trusted-substituters = 
trusted-users = hughdavidson
use-case-hack = true
use-registries = true
use-sqlite-wal = true
use-xdg-base-directories = false
user-agent-suffix = 
warn-dirty = true

[arianvp]

This also affects nix-build.

 $ nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --system 'aarch64-linux'
 
 error: builder for '/nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv' failed with exit code 126;
       last 1 log lines:
       > /nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash: /nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash: cannot execute binary file
       For full logs, run 'nix log /nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv'.
error: 1 dependencies of derivation '/nix/store/1h92zc3pa1v566r2s0za9dfbq9krgwra-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/i01isqxbhsr286da2k6f48bd6i1xssg0-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vnlxxv5fh2hc78hl0whvr25ay5pwwwdd-systemd-255.6.drv' failed to build
arian@Arians-MacBook-Pro nixpkgs % cat /nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv

doesn't work

whilst

 nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --system 'aarch64-linux' --max-jobs 0

does

[misuzu]

I really hate this. What if I want to build a multi-arch derivation? This worked before just fine.

[peterwaller-arm]

Also hit this footgun. Is nix build --rebuild having different logic to select a builder than the default logic? I hit a case where building on the builder worked; but not if I supplied --rebuild.

It's really confusing to try and rebuild a derivation only for it to try and run on the wrong system. I was mislead into thinking the builder was not correctly configured and being silently ignored because the first thing I tried to do was to rebuild a well known derivation.

[roberth]

Discussed in meeting:

Mic92 tried this and it seems to work. Is this error consistently reproducible?

Possible cause: bad connection, in which case a warning is logged, but it is not very prominent and easy to miss.

[Mic92]

I am on nix 2.28 and it don't need to pass --max-jobs 0 to make it use remote builds. However what I think the issue here might be that the root user on this system is not able to ssh into the remote builder. Try sudo nix store info --store ssh://mybuilder to check if the root user is able to access your builder.

In the success case this should look like this:

$ sudo nix store info --store  ssh-ng://nix@graham
Store URL: ssh-ng://
Version: 2.24.14
Trusted: 1

In a broken case it will look like this:

$ sudo nix store info --store ssh-ng://[email protected]
Store URL: ssh-ng://
[email protected]: Permission denied (publickey).
error: failed to start SSH connection to '[email protected]'

I will try the --rebuild flag, because I haven't explicitly tested that.

[misuzu]

In my experience it's usually failing on "local" derivations (preferLocalBuild)

[misuzu]

• Probably related https://github.com/NixOS/nix/issues/9605

[Mic92]

In my experience it's usually failing on "local" derivations (preferLocalBuild)

It would be good if you open new issues instead of grandfathering other issues, if the root cause is not the same with concrete examples to reproduce it.

[Mic92]

Also hit this footgun. Is nix build --rebuild having different logic to select a builder than the default logic? I hit a case where building on the builder worked; but not if I supplied --rebuild.

It's really confusing to try and rebuild a derivation only for it to try and run on the wrong system. I was mislead into thinking the builder was not correctly configured and being silently ignored because the first thing I tried to do was to rebuild a well known derivation.

This is not the same issue as what the original author posted. I opened a new issue for this: https://github.com/NixOS/nix/issues/13114

[misuzu]

Is this it or I need to open a new issue?

% cat /etc/nix/machines
ssh://builder-aarch64-linux-build-box aarch64-linux,armv7l-linux - 64 10 nixos-test,benchmark,big-parallel,kvm - -
% sudo nix store info --store ssh://builder-aarch64-linux-build-box
Store URL: ssh://builder-aarch64-linux-build-box
% cat system.nix
{ config, pkgs, ... }:
{
  boot.loader.grub.device = "nodev";
  fileSystems."/".device = "/dev/sda1";
  services.sshd.enable = true;
}
% nix-build -I nixpkgs=. -I nixos-config=./system.nix '<nixpkgs/nixos>' -A system --system aarch64-linux
evaluation warning: system.stateVersion is not set, defaulting to 25.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
these 27 derivations will be built:
  /nix/store/jya17g2yavj98bps9f0ldvj1m23p3ik7-unit-script-nix-gc-start.drv
  /nix/store/0klvfms8c3iag76kq447aagc3pgvwiw7-unit-nix-gc.service.drv
  /nix/store/72rd5yx73qdhiv94czjm7sxf78vfvc75-nixos-rebuild.drv
  /nix/store/dbjrh0k0wczg18g5pwn0f9f4mf877x5z-nixos-help.drv
  /nix/store/c67plckrv1q38792dzmdf98c3xy151zr-nixos-help.drv
  /nix/store/vvr8mvb7wfffqinzfxymhsgp9bfw1vxy-nixos-version.drv
  /nix/store/7bjylkbsmdpcrms8f0gw6if31yrdqqdh-system-path.drv
  /nix/store/d93bwsww7lv6sn56qk8k3dwhxlz99kj8-dbus-1.drv
  /nix/store/5whm92mah7s8w0nynrzhg2a4vm4vxrpc-X-Restart-Triggers-dbus.drv
  /nix/store/irpwka7pfqv3z1j88b31psfsy3imcn7y-unit-dbus.service.drv
  /nix/store/1pafj2xp0c6q92w0ca3933lwhql2pk8v-user-units.drv
  /nix/store/1w6pbivf7zl72xg4xjm3algx5aig6svq-unit-nix-optimise.service.drv
  /nix/store/ij9lkgi9d3xhihs7kxq6kf9fb5mh1a12-nix.conf.drv
  /nix/store/z8wpx86xxqgyv14vabfv48j1zkdq7wvj-X-Restart-Triggers-nix-daemon.drv
  /nix/store/3r1bwi0jgq7x6fh53xwc052hafvpgjb4-unit-nix-daemon.service.drv
  /nix/store/4349y2s8qys6w7d4yj1ld7sh8wxflalm-unit-container-.service.drv
  /nix/store/9366glcpl7s03rx1lyqvhvv6xprci1ji-tmpfiles.d.drv
  /nix/store/i89ids7dqn28xizgqvcz9wcsdhl3vifz-unit-dbus.service.drv
  /nix/store/fv9i51xgzdjrbgz7xdpn4a7jdi95id4g-X-Restart-Triggers-systemd-tmpfiles-resetup.drv
  /nix/store/yx1cizv2n18jzm5nkr7hp17dah4kba9p-unit-systemd-tmpfiles-resetup.service.drv
  /nix/store/b8x95ng5hhal4z2sr6z4d5s0qh5z66y9-system-units.drv
  /nix/store/52636cz1ldamx0a7fv6nc5bfy6cvamj7-etc.drv
  /nix/store/h4fryprjg3d9nl0y0c2sd524fgginmbf-grub-config.xml.drv
  /nix/store/7qx9ryqzj9gdcvhn0v3npskhjw5n3531-install-grub.sh.drv
  /nix/store/k6rlfzcr02cfq6h5c0h6lgswi4k4if5s-linux-6.12.25-modules.drv
  /nix/store/qwc91f3jilidkhajp7vf05fvh72jvq56-boot.json.drv
  /nix/store/9gnnk0zvrjprzr1wqqdmmqr2xazrlp26-nixos-system-nixos-25.05pre-git.drv
these 16 paths will be fetched (139.03 MiB download, 390.54 MiB unpacked):
  /nix/store/bldk29kcykv4bc70xq9m9ll7b75afcbr-grub-2.12
  /nix/store/iabnap6ibizf0gyv58r6krhv7jwckbvj-initrd-linux-6.12.25
  /nix/store/06lwmzxsyqi3hpvz86g97glpyszrkvii-libblake3-1.8.2
  /nix/store/lrcw50lkh2zf11pnab0z25mdbi6rrmhs-linux-6.12.25
  /nix/store/7ncsapyfs1gdkvbz90sr1aay8pnxrygr-linux-6.12.25-modules-shrunk
  /nix/store/swryx5xyrmxjqsmh794zkr378lnf5w8l-nix-2.28.2
  /nix/store/32ff8ggkhhn93ykw9izql46li9mzxbhm-nix-2.28.2-doc
  /nix/store/x9xhhy881wcsmxpr4plgyddglzfsy2rz-nix-2.28.2-man
  /nix/store/9p3yf5n5lkix6212r81kn3z2ch459cyl-nixos-configuration-reference-manpage
  /nix/store/lp4xwr8znm6dwvf11idmj7z4wx6qxmi1-nixos-manual-html
  /nix/store/k61a152py7hiyffjmvyqz41zd83dhb3v-nixos-option
  /nix/store/0dr58sqwcyq3sjmhb7hvf2f5ad5gjfxp-system-generators
  /nix/store/y9wxhbbfdh8s50imqaxrj09g306dlkyy-system-shutdown
  /nix/store/6z9np1309q0ias7xxbk4mm5fbmn90637-tbb-2021.11.0
  /nix/store/b476kp51wg7ad1aa23bwzbcy1knxj8dj-tbb-2021.11.0-dev
  /nix/store/bha7xkfqm7ykv5jdhc6g6rg545gd20r9-user-generators
copying path '/nix/store/lrcw50lkh2zf11pnab0z25mdbi6rrmhs-linux-6.12.25' from 'https://cache.nixos.org'...
copying path '/nix/store/7ncsapyfs1gdkvbz90sr1aay8pnxrygr-linux-6.12.25-modules-shrunk' from 'https://cache.nixos.org'...
copying path '/nix/store/32ff8ggkhhn93ykw9izql46li9mzxbhm-nix-2.28.2-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/x9xhhy881wcsmxpr4plgyddglzfsy2rz-nix-2.28.2-man' from 'https://cache.nixos.org'...
copying path '/nix/store/9p3yf5n5lkix6212r81kn3z2ch459cyl-nixos-configuration-reference-manpage' from 'https://cache.nixos.org'...
copying path '/nix/store/lp4xwr8znm6dwvf11idmj7z4wx6qxmi1-nixos-manual-html' from 'https://cache.nixos.org'...
copying path '/nix/store/0dr58sqwcyq3sjmhb7hvf2f5ad5gjfxp-system-generators' from 'https://cache.nixos.org'...
copying path '/nix/store/y9wxhbbfdh8s50imqaxrj09g306dlkyy-system-shutdown' from 'https://cache.nixos.org'...
copying path '/nix/store/bha7xkfqm7ykv5jdhc6g6rg545gd20r9-user-generators' from 'https://cache.nixos.org'...
building '/nix/store/vvr8mvb7wfffqinzfxymhsgp9bfw1vxy-nixos-version.drv'...
copying path '/nix/store/bldk29kcykv4bc70xq9m9ll7b75afcbr-grub-2.12' from 'https://cache.nixos.org'...
building '/nix/store/4349y2s8qys6w7d4yj1ld7sh8wxflalm-unit-container-.service.drv'...
copying path '/nix/store/6z9np1309q0ias7xxbk4mm5fbmn90637-tbb-2021.11.0' from 'https://cache.nixos.org'...
copying path '/nix/store/iabnap6ibizf0gyv58r6krhv7jwckbvj-initrd-linux-6.12.25' from 'https://cache.nixos.org'...
error: executing '/nix/store/si7wv0psd4df7bqq71ppcfw7sprlz9j8-bash-5.2p37/bin/bash': Exec format error
error: builder for '/nix/store/vvr8mvb7wfffqinzfxymhsgp9bfw1vxy-nixos-version.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/si7wv0psd4df7bqq71ppcfw7sprlz9j8-bash-5.2p37/bin/bash': Exec format error
       For full logs, run:
         nix log /nix/store/vvr8mvb7wfffqinzfxymhsgp9bfw1vxy-nixos-version.drv
error: 1 dependencies of derivation '/nix/store/7bjylkbsmdpcrms8f0gw6if31yrdqqdh-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9gnnk0zvrjprzr1wqqdmmqr2xazrlp26-nixos-system-nixos-25.05pre-git.drv' failed to build
% nix-build . --no-out-link --fallback -A pkgs.hello --system aarch64-linux --check
this path will be fetched (0.05 MiB download, 0.24 MiB unpacked):
  /nix/store/76plgdn1gbaxh0br5xbc6c7pmc3j6rmk-hello-2.12.1
copying path '/nix/store/76plgdn1gbaxh0br5xbc6c7pmc3j6rmk-hello-2.12.1' from 'https://cache.nixos.org'...
checking outputs of '/nix/store/8ki4hqm7x69bw9inlfjblbx66zmh0m4m-hello-2.12.1.drv'...
error: executing '/nix/store/si7wv0psd4df7bqq71ppcfw7sprlz9j8-bash-5.2p37/bin/bash': Exec format error
error: builder for '/nix/store/8ki4hqm7x69bw9inlfjblbx66zmh0m4m-hello-2.12.1.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/si7wv0psd4df7bqq71ppcfw7sprlz9j8-bash-5.2p37/bin/bash': Exec format error
       For full logs, run:
         nix log /nix/store/8ki4hqm7x69bw9inlfjblbx66zmh0m4m-hello-2.12.1.drv
% nix-build . --no-out-link --fallback -A pkgs.hello --system aarch64-linux --check -j0
checking outputs of '/nix/store/8ki4hqm7x69bw9inlfjblbx66zmh0m4m-hello-2.12.1.drv' on 'ssh://builder-aarch64-linux-build-box'...
copying 0 paths...
/nix/store/76plgdn1gbaxh0br5xbc6c7pmc3j6rmk-hello-2.12.1
% nix --version
nix (Nix) 2.28.2

[Mic92]

@misuzu your remote build works fine, but you are using --system when you should use --eval-system. --system will make your macos machine to pretend it is a Linux machine.

Here is the correct command:

nix-build -I nixpkgs=. -I nixos-config=./system.nix '<nixpkgs/nixos>' -A system --eval-system aarch64-linux

[Mic92]

Ah. Hang on. I actually didn't saw the --rebuild in the original issue post. So my other issue, is just a duplicate. Fixing the title to make it easier to understand what the issue is.

[Mic92]

This also affects nix-build.

 $ nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --system 'aarch64-linux'
 
 error: builder for '/nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv' failed with exit code 126;
       last 1 log lines:
       > /nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash: /nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash: cannot execute binary file
       For full logs, run 'nix log /nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv'.
error: 1 dependencies of derivation '/nix/store/1h92zc3pa1v566r2s0za9dfbq9krgwra-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/i01isqxbhsr286da2k6f48bd6i1xssg0-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vnlxxv5fh2hc78hl0whvr25ay5pwwwdd-systemd-255.6.drv' failed to build
arian@Arians-MacBook-Pro nixpkgs % cat /nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv

Use nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --eval-system 'aarch64-linux' instead.

[misuzu]

you are using --system when you should use --eval-system. --system will make your macos machine to pretend it is a Linux machine.

This makes sense, I didn't even know --eval-system exists. Looks like it was added in 2.20 and the behaviour of --system has changed?

[Mic92]

Looks like it was added in 2.20 and the behaviour of --system has changed?

As far as I remember this always was an issue with --system. Before we had --eval-system, we had to import nixpkgs with a concrete system with import <nixpkgs> { system = "..."; } or use flakes where we have explicit system always set.

[booxter]

Use nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --eval-system 'aarch64-linux' instead.

AFAIU this forces eval on aarch64-linux that is not the current machine. How do I enforce rebuild when the target remote is of the same architecture as my machine where I run nix build from?

[Mic92]

Use nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --eval-system 'aarch64-linux' instead.

AFAIU this forces eval on aarch64-linux that is not the current machine. How do I enforce rebuild when the target remote is of the same architecture as my machine where I run nix build from?

It doesn't to be possible at the moment what this issue is about. I think this would require some refactoring as it's currently implemented in "local-derivation-goal". I don't this ever worked with remote builds. We could for now make a more explicit error.

[Ericson2314]

Notes from the Nix team meeting right now:

@Ericson2314: We think it would be best to fix this after https://github.com/NixOS/nix/issues/5025 is one. The current architecture with the separate l between them makes this sort of bug very likely. A new architecture should make it much easier to prevent bugs like this "by construction".

@edolstra: Could build on DerivationBuilder refactor I just did (#13276) also.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-05-28-nix-team-meeting-minutes-229/65205/1