NixOS
Unhelpful error message when a substituter is missing a dependency
Describe the bug
When trying to fetch the narinfo for a dependency from a substituter fails (i.e. the substituter isn't upholding the store invariants), this isn't reported clearly.
This was first observed with cachix (and doesn't occur reliably there -- maybe its sporadic 502s affect cache service as well?), but can also be reproduced with a synthetic test case.
Steps To Reproduce
rm -rf /tmp/binarycache /tmp/store
drv=$(nix-instantiate '<nixpkgs>' -A hello)
out=$(nix-store -q --outputs $drv)
nix build $drv^out
nix copy $out --to file:///tmp/binarycache
# remove a dependency of hello from the binary cache
rm -v $(grep -lE '^StorePath: .*(glibc|iconv)' /tmp/binarycache/*.narinfo)
nix-store --store /tmp/store --substituters file:///tmp/binarycache?trusted=1 -r $out --narinfo-cache-positive-ttl 1 --narinfo-cache-negative-ttl 1
The final command will fail something like this:
this path will be fetched (0.05 MiB download, 0.22 MiB unpacked):
/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1
don't know how to build these paths:
/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36
error: build of '/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1' failed
If the drv is copied to the chroot store and realised instead of its output path, the behaviour combines somewhat oddly, claiming that it doesn't know how to build the dependency and yet starting to do so anyway:
$ nix-store --store /tmp --substituters file:///tmp/binarycache?trusted=1 -r /nix/store/7s3p1q56abzik34w93kw7jx37l37ycg8-hello-2.12.1.drv --narinfo-cache-positive-ttl 1 --narinfo-cache-negative-ttl 1 --no-fallback
this path will be fetched (0.05 MiB download, 0.22 MiB unpacked):
/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1
don't know how to build these paths:
/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36
building '/nix/store/1crz16gbhsxjxkyp2zxjf5y7a224m4j7-bash52-033.drv'...
building '/nix/store/0m4y3j4pnivlhhpr5yqdvlly86p93fwc-busybox.drv'...
[...]
Expected behavior
A warning when a path that's expected in a binary cache (based on the store invariant "all dependencies should be there"), ideally with details on how getting the narinfo failed.
Metadata
nix (Nix) 2.26.1
Additional context
It might be possible to trigger the same kind of failure without breaking the "all dependencies are present" invariant if through a somewhat convoluted combination of GC operations:
- The narinfo for the dependent (hello) is present in the client's narinfo cache
- The narinfo for the dependency (glibc or iconv) is not in the client's narinfo cache (glibc/iconv was already present on the client when the dependent was last queried)
- The narinfo for the dependency is not present in the binary cache due to a GC.
Checklist
- [x] checked latest Nix manual (source)
- [x] checked open bug issues and pull requests for possible duplicates
Add :+1: to issues you find important.
Discussed in meeting:
don't know how to build these paths:
This comes from libmain/shared.cc, which gets info from Store::queryMissing through various CLI code paths.
Unfortunately this code is disjoint from the Worker scheduler which seems to have more advanced logic for this kind of situation.
It might be possible to just improve the error message to account for this possibility. Before doing so, it would be good to figure out which causes can be distinguished for derivations to end up in the unknown StorePathSet. It's a catch-all for all "bad news" currently.
We discussed this a little bit more today. @edolstra, who wasn't there last time, is intrigued by the "have Worker handle --dry-run, so we can delete Store::queryMissing" idea, also.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2025-05-28-nix-team-meeting-minutes-229/65205/1
