NixOS
Nix daemon crashes with assertion failure when building multiple mlibc cross-compiled packages
Describe the bug
Nix daemon crashes with assertion failure when building multiple cross-compiled packages for aarch64-linux-mlibc target in parallel. The crash occurs in the BinaryCacheStore's callback mechanism with the error: Assertion '!prev' failed.
Steps To Reproduce
-
Run the following command:
nix build --no-link -L github:NixOS/nixpkgs/pull/371092/merge#{pkgsCross.aarch64-linux-mlibc.ed,pkgsCross.aarch64-linux-mlibc.vim,pkgsCross.aarch64-linux-mlibc.tmux,pkgsCross.aarch64-linux-mlibc.git} -
Observe the error:
error: Nix daemon disconnected unexpectedly (maybe it crashed?) -
Check system logs to see the assertion failure and stack trace
Expected behavior
The packages should build successfully without crashing the nix daemon. Cross-compilation for mlibc targets should work reliably when building multiple packages in parallel.
Metadata
nix-env (Nix) 2.28.3
Additional context
Stack Trace
nix-daemon: /nix/store/XHD38P32PVBZQMI0ZQ1WICLJY5B7YQDV-nix-util-2.28.3-dev/include/nix/util/callback.hh:45: void nix::Callback<T>::rethrow(const std::__exception_ptr::exception_ptr&) [with T = std::optional<std::__cxx11::basic_string<char> >]: Assertion `!prev' failed.
Stack trace:
0# 0x000000000048F74B in nix-daemon
1# 0x00007FD49D8BC20A in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
2# 0x00007FD49D8BC275 in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
3# 0x00007FD49E092358 in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
4# nix::Store::queryPathInfo(nix::StorePath const&, nix::Callback<nix::ref<nix::ValidPathInfo const> >) in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
5# nix::Store::queryPathInfo(nix::StorePath const&) in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
6# nix::Store::querySubstitutablePathInfos(std::map<nix::StorePath, std::optional<nix::ContentAddress>, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, std::optional<nix::ContentAddress> > > > const&, std::map<nix::StorePath, nix::SubstitutablePathInfo, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::SubstitutablePathInfo> > >&) in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
7# 0x00007FD49E1D4FBE in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
8# 0x00007FD49E1DC712 in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
9# nix::ThreadPool::doWork(bool) in /nix/store/319hpn6dffzslywqhvjcwhs8xj822rdl-nix-util-2.28.3/lib/libnixutil.so
10# 0x00007FD49D8E86D3 in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
11# 0x00007FD49D672D02 in /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
12# 0x00007FD49D6F23AC in /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
Key observations:
- Crash occurs specifically when building multiple
aarch64-linux-mlibccross-compiled packages in parallel - Building individual packages works fine
- Crash happens in
BinaryCacheStore::queryPathInfoUncachedwhen querying substitutable paths - Possible race condition in the callback handling mechanism
- Issue discovered while testing nixpkgs PR https://github.com/NixOS/nixpkgs/pull/371092
Workarounds:
- Build packages individually instead of all at once
- Use
--option substituters ""to disable substituters - Build with
--max-jobs 1(though this doesn't always prevent the crash)
System information:
- Platform: Linux x86_64
- NixOS version: unstable
- The crash is reproducible and happens immediately when the command is run
Checklist
- [x] checked latest Nix manual (source)
- [x] checked open bug issues and pull requests for possible duplicates
Add :+1: to issues you find important.
Randomly ran into the same assertion failure on master, so at least >= 2.28, <= master are affected.
@xokdvium can you elaborate on how you triggered it? And provide a stack trace if possible.
@xokdvium can you elaborate on how you triggered it
I think it was rather random, just when rebuilding the system.
Unfortunately the coredump isn't available, so I don't have a stacktrace atm. Will try to reproduce.
I seem to have encountered a similar error:
- Clone the repo
git clone -b nix-crash https://github.com/wind-mask/zellij-autolock.git - Run
nix build
Metadata
nix-env --version
nix-env (Nix) 2.32.1
Additional context
I'm using Nix on Arch.
╭──────────────────┬────────────────────────────────────────────────────────╮
│ kernel-name │ Linux │
│ nodename │ arch │
│ kernel-release │ 6.17.1-2-cachyos │
│ kernel-version │ #1 SMP PREEMPT_DYNAMIC Mon, 06 Oct 2025 23:26:58 +0000 │
│ machine │ x86_64 │
│ operating-system │ GNU/Linux │
╰──────────────────┴────────────────────────────────────────────────────────╯
The crash is consistently reproducible on my system.
log
10月 18 22:19:24 arch nix-daemon[344302]: terminating due to unexpected unrecoverable internal error: Assertion '!prev' failed in void nix::Callback<T>::rethrow(const std::__except>
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: terminating due to unexpected unrecoverable internal error: Assertion '!prev' failed in void nix::Callback<T>::rethrow(const std::__except>
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::FileTransferError: error: download of 'https://cache.nixos.org/yaq3pczkc6dvrqksxgydzdip5yhzb78x.narinfo' was interrupted
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::FileTransferError: error: download of 'https://cache.nixos.org/ylrmv07wyn0vz80prvn0i6i5aikh4mc7.narinfo' was interrupted
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::FileTransferError: error: download of 'https://cache.nixos.org/yaq3pczkc6dvrqksxgydzdip5yhzb78x.narinfo' was interrupted
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::FileTransferError: error: download of 'https://cache.nixos.org/ylrmv07wyn0vz80prvn0i6i5aikh4mc7.narinfo' was interrupted
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: terminating due to unexpected unrecoverable internal error: Assertion '!prev' failed in void nix::Callback<T>::rethrow(const std::__except>
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: terminating due to unexpected unrecoverable internal error: Assertion '!prev' failed in void nix::Callback<T>::rethrow(const std::__except>
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::FileTransferError: error: download of 'https://cache.nixos.org/c2rp1lkhkv2v57dqw9wnbmg384086kq9.narinfo' was interrupted
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::Interrupted: error: interrupted by the user
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::FileTransferError: error: download of 'https://cache.nixos.org/c2rp1lkhkv2v57dqw9wnbmg384086kq9.narinfo' was interrupted
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: Exception: nix::Interrupted: error: interrupted by the user
10月 18 22:19:24 arch nix-daemon[344302]: Stack trace:
10月 18 22:19:24 arch nix-daemon[344302]: 0# 0x00000000000CCACA in nix-daemon
10月 18 22:19:24 arch nix-daemon[344302]: 1# 0x00000000000B3C8C in /usr/lib/libstdc++.so.6
10月 18 22:19:24 arch nix-daemon[344302]: 2# std::unexpected() in /usr/lib/libstdc++.so.6
10月 18 22:19:24 arch nix-daemon[344302]: 3# 0x00000000000C5E53 in /usr/lib/libnixutil.so.2.32.1
10月 18 22:19:24 arch nix-daemon[344302]: 4# __wrap___assert_fail in nix-daemon
10月 18 22:19:24 arch nix-daemon[344302]: 5# 0x0000000000083293 in /usr/lib/libnixstore.so.2.32.1
10月 18 22:19:24 arch nix-daemon[344302]: 6# 0x00000000000D09A7 in /usr/lib/libnixstore.so.2.32.1
10月 18 22:19:24 arch nix-daemon[344302]: 7# 0x00000000001FB3CD in /usr/lib/libnixstore.so.2.32.1
10月 18 22:19:24 arch nix-daemon[344302]: 8# 0x0000000000134E8E in /usr/lib/libnixstore.so.2.32.1
10月 18 22:19:24 arch nix-daemon[344302]: 9# 0x00000000000B94CC in /usr/lib/libnixstore.so.2.32.1
10月 18 22:19:24 arch nix-daemon[344302]: 10# 0x00000000001002F4 in /usr/lib/libstdc++.so.6
10月 18 22:19:24 arch nix-daemon[344302]: 11# 0x00000000000A80D6 in /usr/lib/libc.so.6
10月 18 22:19:24 arch nix-daemon[344302]: 12# 0x000000000013B01C in /usr/lib/libc.so.6
10月 18 22:19:24 arch nix-daemon[344302]: 0# 0x00000000000CCACA in nix-daemon
1# 0x00000000000B3C8C in /usr/lib/libstdc++.so.6
2# std::unexpected() in /usr/lib/libstdc++.so.6
3# 0x00000000000C5E53 in /usr/lib/libnixutil.so.2.32.1
4# __wrap___assert_fail in nix-daemon
5# 0x0000000000083293 in /usr/lib/libnixstore.so.2.32.1
6# 0x00000000000D09A7 in /usr/lib/libnixstore.so.2.32.1
7# 0x00000000001FB3CD in /usr/lib/libnixstore.so.2.32.1
8# 0x0000000000134E8E in /usr/lib/libnixstore.so.2.32.1
9# 0x00000000000B94CC in /usr/lib/libnixstore.so.2.32.1
10# 0x00000000001002F4 in /usr/lib/libstdc++.so.6
11# 0x00000000000A80D6 in /usr/lib/libc.so.6
12# 0x000000000013B01C in /usr/lib/libc.so.6
@wind-mask are you still able to replicate? I'm on Nix 2.32.4 and cannot replicate, I get this:
❯ nix build
error: Cannot build '/nix/store/cgidl5jhxkqdmsi4k9z2rrhp4rg09f0h-zellij-autolock.drv'.
Reason: builder failed with exit code 101.
Output paths:
/nix/store/ynvwjhp5ff3b0wa4w0djni1g1245yfww-zellij-autolock
Last 25 log lines:
>
> Could not find directory of OpenSSL installation, and this `-sys` crate cannot
> proceed without this knowledge. If OpenSSL is installed and this crate had
> trouble finding it, you can set the `OPENSSL_DIR` environment variable for the
> compilation process.
>
> Make sure you also have the development packages of openssl installed.
> For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.
>
> If you're in a situation where you think the directory *should* be found
> automatically, please open a bug at https://github.com/rust-openssl/rust-openssl
> and include information about your system as well as this message.
>
> $HOST = x86_64-unknown-linux-gnu
> $TARGET = x86_64-unknown-linux-gnu
> openssl-sys = 0.9.110
>
>
> It looks like you're compiling on Linux and also targeting Linux. Currently this
> requires the `pkg-config` utility to find OpenSSL but unfortunately `pkg-config`
> could not be found. If you have OpenSSL installed you can likely fix this by
> installing `pkg-config`.
>
>
> warning: build failed, waiting for other jobs to finish...
For full logs, run:
nix log /nix/store/cgidl5jhxkqdmsi4k9z2rrhp4rg09f0h-zellij-autolock.drv
@siraben I've added buildInputs = [ pkgs.openssl pkgs.pkg-config ]. Now you can try again:
Clone the repo git clone -b nix-crash https://github.com/wind-mask/zellij-autolock.git
Run nix build
It succeeded on GitHub Action: https://github.com/wind-mask/zellij-autolock/actions/runs/19534922443
On my computer, if I run nix build directly, my session still crashes, but it's not the nix-daemon crashing (and there's no coredump either). It seems to be some kind of concurrency-related resource issue? When I limit resources with nix build --cores 2 --max-jobs 1, it doesn't crash.
version
nix --version
nix (Nix) 2.32.4
@wind-mask weird, it builds on my computer, no crash
$ nix build
$ readlink -f ./result
/nix/store/3k9vg77i3abknrqjklrzk77ii32b4wcm-zellij-autolock
@siraben Then it appears to be just some kind of resource limitation issue, such as OOM / Resource Exhaustion. The original internal crash might no longer exist now.
The underlying issue could probably be https://github.com/NixOS/nix/pull/14418. It's an intermittent issue when nix tries to re-enqueue a file transfer when it was interrupted or otherwise failed. Maybe an intermittent curl failure could cause this, but I was able to reproduce this somewhat consistently when interrupting an upload to a binary cache.