NixOS
Assertion failure in `outputsSubstitutionTried`
Describe the bug
When running nix develop it stops with error: Nix daemon disconnected unexpectedly (maybe it crashed?). In daemon logs I can see
$ journalctl -u nix-daemon.service -n2
Feb 26 19:46:26 voyager nix-daemon[1567]: accepted connection from pid 3366293, user t4ccer (trusted)
Feb 26 19:46:57 voyager nix-daemon[3366304]: nix-daemon: src/libstore/build/derivation-goal.cc:326: void nix::DerivationGoal::outputsSubstitutionTried(): Assertion `false' failed.
I'm sorry for not being able to provide more details because after waiting ~30 minutes it stopped happening (probably remote builder and binary cache caught up and substituted before hitting this) and I cannot reproduce it myself now. I hope that it still may be a valuable report.
Expected behavior
It doesn't happen.
nix-env --version output
nix-env (Nix) 2.19.3
Additional context
Add any other context about the problem here.
Priorities
Add :+1: to issues you find important.
This assertion was introduced here https://github.com/NixOS/nix/pull/6312/files#diff-5784d1ad7c22601c75b86872534049bea5173377de5d61b7446452beabc16d3fR323
I can reproduce this deterministically on a private project. nix develop would fail with:
nix-daemon: src/libstore/unix/build/derivation-goal.cc:335: void nix::DerivationGoal::outputsSubstitutionTried(): Assertion false' failed.`
Before failing, the nix develop shows
warning: ignoring substitute for '/nix/store/j1hbm4czfvv8jp6bm1qzxj8ykmxx5isb-gstreamer-1.22.9' from 's3://REDACTED_PRIVATE_NIX_CACHE', as it's not signed by any of the keys in
nix develop works well for other projects (that dont have the substitution issue).
This happens with
➜ nix --version
nix (Nix) 2.22.1
I can work around it by disabling the cache with bad signatures `nix develop --option substituters 'https://cache.nixos.org/' works but sadly this is one of the project with a long time build. I guess I could fix those signatures as well ^^''
not sure if I messed up my previous test or if 2.23.1 regressed but with 2.23.1 I had it again nix-daemon: src/libstore/build/derivation-goal.cc:303: void nix::DerivationGoal::outputsSubstitutionTried(): Assertion false' failed.`. Patching the assert out works though
After updating flake.lock on my nixos configuration, while on nix version 2.18.4 (on latest nixos) I encounter the same error now. Tried with nix 2.23.1, 2.22 and 2.21 and am seeing the same result.
Still happening with 2.23.3 on Arch. No issue after downgrading to 2.15.0.
Happening with 2.27.1 while pulling from a github.com/zhaofengli/attic cache
I just run into this with 2.26.3
I ran nix store gc, then on the next build I got an error about download-buffer-size being too small. After I fixed that I run into this error. I'm using atticd for a few months and ncps since yesterday.
A few minutes later:
It started working again. The only change I'm aware I made since is, setting builders-use-substitutes = true
I see this issue happening on Debian Sid with Nix install via apt.
nix --version
nix (Nix) 2.24.12
nix build -L --option binary-caches 'https://ngi.cachix.org/' --option trusted-public-keys 'ngi.cachix.org-1:n+CAL72ROC3qQuLxIHpV+Tw5t42WhXmMhprAGkRSrOw=' --extra-experimental-features nix-command -f default.nix
[1/0/3608 built, 0/70 copied (0.0/1.6 MiB), 0.0/0.1 MiB DL] building bash-5.2.tar.gznix: src/libstore/build/derivation-goal.cc:306: nix::Goal::Co nix::DerivationGoal::outputsSubstitutionTried(): Assertion `false' failed.
Aborted (core dumped)
Building the same thing on Debian Sid with Nix installed via DetSys installer fails on this
nix --version
nix (Nix) 2.27.1
nix-build: ../build/derivation-goal.cc:293: nix::Goal::Co nix::DerivationGoal::haveDerivation(): Assertion `false' failed.
Aborted (core dumped)
This works (nix installed via apt)
nix-build --option binary-caches 'https://cache.nixos.org/ https://ngi.cachix.org/' --option trusted-public-keys 'cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ngi.cachix.org-1:n+CAL72ROC3qQuLxIHpV+Tw5t42WhXmMhprAGkRSrOw= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=' default.nix
Difference to previous commands is added cache and key for https://cache.nixos.org/.
Got this too with:
$ nix --version
nix (Nix) 2.24.12
$ cat flake.lock | jq .nodes.nixpkgs
{
"locked": {
"lastModified": 1702272962,
"narHash": "sha256-D+zHwkwPc6oYQ4G3A1HuadopqRwUY/JkMwHz1YF7j4Q=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "e97b3e4186bcadf0ef1b6be22b8558eab1cdeb5d",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixpkgs-unstable",
"repo": "nixpkgs",
"type": "github"
}
}
When uncommenting these lines of code in my config:
(lib.mkIf true (let
kodi = (pkgs.kodi-wayland.withPackages (kodiPkgs: [
kodiPkgs.jellyfin
]));
in {
users.extraUsers.kodi.isNormalUser = true;
services.cage.user = "skarabox";
services.cage.program = "${kodi}/bin/kodi-standalone";
services.cage.enable = true;
}))
Indeed, re-enabling the substituter makes it work.
There has been some restructuring of the relevant code recently. Would be helpful to confirm this issues with a more recent version. (past 2.27.1?) Since #6312 ?
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2025-05-04-nix-team-meeting-minutes-230/65206/1
@tomberek I just hit this issue on our CI (the original failure on 1.22, and the new one (nix-build: ../build/derivation-goal.cc:313: nix::Goal::Co nix::DerivationGoal::haveDerivation(): Assertion `false' failed.) on 1.29.1. This is using an attic cache and I had forcefully terminated the previous upload, in case that matters. It seems reproducible so far.
I just noticed I get "don't know how to build these paths" right before the assert, not sure if it's normal.
Details
this derivation will be built:
/nix/store/ck4cr2wdh1qr17hb5yn2r3cp82w89706-torch-2.7.1.drv
these 5 paths will be fetched (0.00 MiB download, 385.47 MiB unpacked):
/nix/store/95jcz46l86b18ai6407zzvw31yinpn3g-cudnn-9.8.0.87-dev
/nix/store/kqcav0vdmpm52kl39zj1l6ly3i7lypk6-libcublas-12.8.4.1-dev
/nix/store/cqd1yh025ay8kpsdnfv1qg1dw86czqba-libcufft-11.3.3.83-dev
/nix/store/kjxhr2gs5a4h59mpmhswhvk7n05816aq-libcusolver-11.7.3.90-dev
/nix/store/xcr7hv5vybvmzc06v1ykj9nx659mb0w8-libcusolver-11.7.3.90-lib
don't know how to build these paths:
/nix/store/3m62ns8i686lk81c7g6jlp6zxbzbq5hv-libcublas-12.8.4.1-lib
/nix/store/8g057nl8sy4i88hg5ssyqz376jy6w3xk-cudnn-9.8.0.87-static
/nix/store/h5pakqdgzabil7j6rjhrw0b1fq6y43j6-cudnn-9.8.0.87-lib
copying path '/nix/store/cqd1yh025ay8kpsdnfv1qg1dw86czqba-libcufft-11.3.3.83-dev' from 'https://attic-cache-here/'...
nix-build: ../build/derivation-goal.cc:313: nix::Goal::Co nix::DerivationGoal::haveDerivation(): Assertion `false' failed.
Aborted
$ echo /nix/store/*cudnn-9.8.0.87*
/nix/store/savcvzmq96x7fj4xhimshki1lkldzlk5-cudnn-9.8.0.87 /nix/store/v2fbijshri4cy1l2f5h1j8gqx20nva99-cudnn-9.8.0.87.drv
$ nix show-derivation /nix/store/v2fbijshri4cy1l2f5h1j8gqx20nva99-cudnn-9.8.0.87.drv
....
"name": "cudnn-9.8.0.87",
"outputs": {
"dev": {
"path": "/nix/store/95jcz46l86b18ai6407zzvw31yinpn3g-cudnn-9.8.0.87-dev"
},
"lib": {
"path": "/nix/store/h5pakqdgzabil7j6rjhrw0b1fq6y43j6-cudnn-9.8.0.87-lib"
},
"out": {
"path": "/nix/store/savcvzmq96x7fj4xhimshki1lkldzlk5-cudnn-9.8.0.87"
},
"static": {
"path": "/nix/store/8g057nl8sy4i88hg5ssyqz376jy6w3xk-cudnn-9.8.0.87-static"
}
....
If I look in the store, some of these derivations were built, but most are missing outputs. However if I try to build it with nix-build, I only get the main output and not the other ones. I then tried building it in nix repl (:b (import /path/to/drv)) which worked, and subsequently fixed the build that was broken. I'll give this another try tomorrow to see if I can gather more details.
Looks like the 2.30 release removed the feature that was causing the assertion failure in #13176