"error: cannot link '/nix/store/.tmp-link' to '/nix/store/.links/...': File exists"

[grahamc]

Describe the bug

Remote building from Nix 2.11.0 on a Mac to an aarch64 Linux machine running 2.8.1, where the remote builder has auto-optimise-store = true, occasionally we're seeing errors on the remote:

error: cannot link '/nix/store/.tmp-link-13949-1705072255' to '/nix/store/.links/09vd...mxd': File exists

this ends up failing the "substitution" and requires retrying. After trying many times, it eventually succeeds.

Expected behavior

1. Optimising the store should not fail if racing to create the link fails.
2. A remote build shouldn't fail if optimising fails.

nix-env --version output

[root@nix-builder:~]# nix-env --version
nix-env (Nix) 2.8.1

$ nix-env --version
nix-env (Nix) 2.11.0

Additional context

I don't believe any changes have been made to Nix since 2.8.1 that impacted auto-optimisation.

[esoterick]

I have had this issue for a while on aarch64 macos... or at least something similar to this issue. It only seems to occur when I have a large set of updates to apply. I also just finished updating nix and my flakes so reproduction is likely not possible on my work machine until upstream flakes have more updates and I update again.

With that said if any other info is needed I'd be happy to help and provide it.

~ main
λ while ! darwin-rebuild build --flake .; echo "running again"; end
building the system configuration...
error: cannot link '/nix/store/.tmp-link-30976-355646950' to '/nix/store/.links/0p6h759asm5aw4d11j0mfbd73wfnnszdp83aggbb64spvhld6scp': File exists
error: some substitutes for the outputs of derivation '/nix/store/wdc9847w1c0hf4r24xdm1vbrm4xfcqmi-ruby-2.7.6.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 1 dependencies of derivation '/nix/store/73678nhsnvz6q7cmf6c9ypn46mw8v4nq-neovim-ruby-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vqn2ddzlqqlwdrb414bf6llm5k419kqh-neovim-0.8.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jv9c15jp43dcpv2h3y5ydyq94vn2k15x-home-manager-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kra7m82rs042ib1zliw87d5jj1ag067h-home-manager-fonts.drv' failed to build
error: 1 dependencies of derivation '/nix/store/m7vc4qgmgb39hw565h8a00lidgmnpd5y-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yj1mhcx42fjqpsg814g9ilc8k3s1cvnb-man-paths.drv' failed to build
error: 1 dependencies of derivation '/nix/store/s60gbpgxqfxsm21dgrvbgz125jcknnfv-neovim-0.8.0-fish-completions.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y0058m20lplqcm5v6c866bsyyszfxn9f-system-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/a5ghrh4xiak7zg37gv1sfbfah9rsfm6a-darwin-system-22.11.20221108.295778a+darwin4.cfc0125.drv' failed to build
running again
building the system configuration...
error: cannot link '/nix/store/.tmp-link-31147-65897529' to '/nix/store/.links/02kyn2r2hwd6jlbgz61b5b4k9i0pxrlkym867s4js9srcfwhdh7j': File exists
error: some substitutes for the outputs of derivation '/nix/store/pirbrgvn26zmh0jdcf3xfjdqqhh0jn30-neovim-unwrapped-0.8.0.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 1 dependencies of derivation '/nix/store/vqn2ddzlqqlwdrb414bf6llm5k419kqh-neovim-0.8.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jv9c15jp43dcpv2h3y5ydyq94vn2k15x-home-manager-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kra7m82rs042ib1zliw87d5jj1ag067h-home-manager-fonts.drv' failed to build
error: 1 dependencies of derivation '/nix/store/m7vc4qgmgb39hw565h8a00lidgmnpd5y-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yj1mhcx42fjqpsg814g9ilc8k3s1cvnb-man-paths.drv' failed to build
error: 1 dependencies of derivation '/nix/store/s60gbpgxqfxsm21dgrvbgz125jcknnfv-neovim-0.8.0-fish-completions.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y0058m20lplqcm5v6c866bsyyszfxn9f-system-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/a5ghrh4xiak7zg37gv1sfbfah9rsfm6a-darwin-system-22.11.20221108.295778a+darwin4.cfc0125.drv' failed to build
running again
building the system configuration...
error: cannot link '/nix/store/.tmp-link-31559-489862555' to '/nix/store/.links/0dlm8jl3p1b00acfaxgx43bgvz1xyj2zzx22nld49rm2pqs2kd6f': File exists
error: some substitutes for the outputs of derivation '/nix/store/w2r92c0gcp54d5kwy43s8976jkbk5byv-fish-3.5.1.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 1 dependencies of derivation '/nix/store/5jzlv5yn2chz519j4dva09qmgqzczkb8-1password-cli-2.7.3-fish-completions.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jv9c15jp43dcpv2h3y5ydyq94vn2k15x-home-manager-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kra7m82rs042ib1zliw87d5jj1ag067h-home-manager-fonts.drv' failed to build
error: 1 dependencies of derivation '/nix/store/m7vc4qgmgb39hw565h8a00lidgmnpd5y-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yj1mhcx42fjqpsg814g9ilc8k3s1cvnb-man-paths.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y0058m20lplqcm5v6c866bsyyszfxn9f-system-applications.drv' failed to build
error: 1 dependencies of derivation '/nix/store/a5ghrh4xiak7zg37gv1sfbfah9rsfm6a-darwin-system-22.11.20221108.295778a+darwin4.cfc0125.drv' failed to build
running again
building the system configuration...
error: cannot link '/nix/store/.tmp-link-32406-59810143' to '/nix/store/.links/1x9ipagqb7l5kfhcx1266637cx64ma0n8qvlr4rpxwqwwsnyigbd': File exists
running again
building the system configuration...

~ main 16m15s
λ

[m9tdev]

I have the same issue on x86_64-darwin using nix-darwin with:

nix.settings.auto-optimise-store = true;
services.nix-daemon.enable = true;

I've turned off auto-optimise-store and deleted /nix/store/.links because the "cannot link" errors were making it unusable for me unfortunately.

[Mayeu]

I have had this issue for a while on x86_64-darwin as well. With my main nix-darwin config, but also any of my local {flake,default,shell}.nix files.

My workaround is to relaunch the command again multiple time to make the problem vanish.

I do have auto-optimise as well as various keep option setup:

auto-optimise-store = true
keep-outputs = true
keep-derivations = true
keep-failed = false
keep-going = true

Example of this behaviour with direnv, I had to launch it 3 times to make it through:

➜ direnv reload
direnv: loading ~/Documents/code/nixos-server/.envrc
direnv: using flake
evaluating derivation 'path:/Users/m/Documents/code/nixos-server#devShells.x86_64-darwin.default'direnv: ([/run/current-system/sw/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.
error: cannot link '/nix/store/.tmp-link-83762-706854433' to '/nix/store/.links/1j1za36sy5z7q9rbysrgmpfr7p9m273w9w7dh4i40ighxagh7qg5': File exists
error: cannot link '/nix/store/.tmp-link-83762-775054511' to '/nix/store/.links/1h15xhks04b6bkf9bwpdfsl23wlfs2crq4bg3h7daqd6v7665avm': File exists
error: cannot link '/nix/store/.tmp-link-83762-1213554830' to '/nix/store/.links/042sc4myblqp3jijnxc7q9q6mk6vy2pv17fpk1vxy1wk8crc8m5y': File exists
error: cannot link '/nix/store/.tmp-link-83762-905603765' to '/nix/store/.links/1ywhb1yz57ldknhvvz1qxag44cxjzylspk7hwba5x4cywm2cf72i': File exists
error: cannot link '/nix/store/.tmp-link-83762-2096820833' to '/nix/store/.links/002k633c4v2mw48l395gslnzims1jbpcs4cf1lavinb5xkl3v2rk': File exists
error: cannot link '/nix/store/.tmp-link-83762-847662844' to '/nix/store/.links/0k5dn84cljciy9mrlf366hbmk87szbag2812rmbfzdjf7namid3w': File exists
error: some substitutes for the outputs of derivation '/nix/store/5mddx5crmvrfz6zxg8jxjyr63pl0sk8n-git-2.36.2.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: some substitutes for the outputs of derivation '/nix/store/1l1y5nvy474xaiw3j817s681rvr6vpg0-qemu-7.1.0.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: 2 dependencies of derivation '/nix/store/bbcw5fscinvz5giamkzgxgrnj9sij1ch-nix-shell-env.drv' failed to build
direnv: nix-direnv: renewed cache
direnv: export ~XDG_DATA_DIRS

1m 48.2s
➜ direnv reload
direnv: loading ~/Documents/code/nixos-server/.envrc
direnv: using flake
[0/1 built] direnv: ([/run/current-system/sw/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.
error: cannot link '/nix/store/.tmp-link-84114-1569031621' to '/nix/store/.links/05xsyqwzqdzdcsa2qfjh4q324zq14bl8msmzcmdsgknrkl208r3x': File exists
direnv: nix-direnv: renewed cache
direnv: export [...]

35.0s
➜ direnv reload
direnv: loading ~/Documents/code/nixos-server/.envrc
direnv: using flake
direnv: ([/run/current-system/sw/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.
direnv: nix-direnv: renewed cache
direnv: export [...]

[CorbanR]

Thanks to someone pointing out this bug to me on Elements #Nix on MacOS channel. I have had this exact issue for quite a long time. My work around initially has been to wrap my build command in a ruby script that captures they error (through regex matching), and just continually retry until the build succeeds.

The suggested work around I from the Element chat, was to disable automatic store optimization auto-optimise-store = false and either run nix store optimise manually, or via a scheduled launchd. I will try the latter and report back if it solved my issue.

• system: "aarch64-darwin"
• host os: Darwin 21.6.0, macOS 12.6.1
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.11.0
• nixpkgs: /nix/store/g74cwvm76h14z1jh7gs1bvbqz2m6wc6l-source

[chreekat]

GHC CI is also affected: https://gitlab.haskell.org/ghc/ghc/-/issues/22860

[CorbanR]

Thanks to someone pointing out this bug to me on Elements #Nix on MacOS channel. I have had this exact issue for quite a long time. My work around initially has been to wrap my build command in a ruby script that captures they error (through regex matching), and just continually retry until the build succeeds.

The suggested work around I from the Element chat, was to disable automatic store optimization auto-optimise-store = false and either run nix store optimise manually, or via a scheduled launchd. I will try the latter and report back if it solved my issue.

* system: `"aarch64-darwin"`

* host os: `Darwin 21.6.0, macOS 12.6.1`

* multi-user?: `yes`

* sandbox: `no`

* version: `nix-env (Nix) 2.11.0`

* nixpkgs: `/nix/store/g74cwvm76h14z1jh7gs1bvbqz2m6wc6l-source`

Setting auto-optimise-store = false and running nix store optimise manually seems to have fixed the issue for me. I no longer receive this error. (On OSX)

[ryanbooker]

I get this all the time (again nix-darwin with auto-optimise enabled), especially on slower machines. Often you can get around it by running the build multiple times.

Using --fallback on nix build is another work around.

[maxammann]

Also can confirm, has this in my nix config:

  nix.extraOptions = ''
    auto-optimise-store = true
  '';

[takeda]

Can confirm I also started seeing these errors after enabling auto-optimise-store

nix 2.13.3 on darwin.

Unlike original reporter my builds are happening locally.

[j-baker]

I'm seeing the same thing.

[tfc]

I'm also running into this on an x86_64 macbook. Using the --fallback flag works.

[bryanhonof]

I'm experiencing the same behavior with auto-optimise-store set to true on aarch64-darwin.

[zdc13]

For me, neither nix optimise, nor nix-collect-garbage helped. The --fallback option wasn't available (AFAIT) on home-manager. Just re-running the build command a bunch of times eventually had it working.

I have these options:

keep-derivations = false
auto-optimise-store = false

[KristianBalaj]

I'm experiencing the same when auto-optimise-store = true being on x86_64-darwin

[johnnywalker]

I encountered the same issue when using auto-optimise-store = true on x86_64-darwin. After disabling, I had to restart the nix-daemon as well.

sudo rm -rf /nix/store/.links
sudo launchctl stop org.nixos.nix-daemon
sudo launchctl start org.nixos.nix-daemon

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-can-i-optimize-a-store-that-looks-like-this/44002/7

[carlthome]

Does anyone have a launchd workaround to share that works in e.g. nix-darwin or home manager?

[reckenrode]

If you set nix.optimise.automatic = true; in your nix-darwin config, it will set up a launchd service to optimize the store.

https://daiderd.com/nix-darwin/manual/index.html#opt-nix.optimise.automatic

[lunik1]

I've also been seeing this when optimising the store manually with nix store optimise.

[georgyo-js]

I can hit this bug on linux with Nix 2.24.10 when it built staticly with mulc, but not when it is built with glibc.

So this is not a darwin specific bug. My reproducer use a static nix to bootstrap nix is

count=0
while (bwrap \
  --dev-bind / / \
  --tmpfs /nix \
  -- \
  <path to a statically built nix> \
  build \
  --no-link \
  --option auto-optimise-store true \
  /nix/store/yzbchf7s46gawyym6i0bdwb38jar4fb0-nix-2.24.10);
do count=$(( $count + 1 ))
done
echo $count

After running it three times I hit this bug with counts of 562, 261, and 6003. It produces errors that look like following:

filesystem error: cannot create hard link: File exists [nix/store/.links/1p1vj24i0kzhh6sm650mxa2slqy4y201c1ddqz70sv4a0d456fmz] [/nix/store/.tmp-link-45528-2095319450]

Initally I hit this on XFS filesystem, but this reproducer uses tmpfs proving its not a filesystem specific issue.

[roberth]

Triaged in meeting: This is probably a race condition in the store optimizing routine.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-05-04-nix-team-meeting-minutes-230/65206/1