Temporary build directories not cleaned up because they are not empty

[andir]

Describe the bug

When cancelling a build (started with nix-build) by pressing Ctrl-C the temporary build directories are not (always?) cleaned up. Nix tries to remove some directories but fails as they aren't empty. This ends up costing RAM (tmpfs) or disk space when no manual cleanup is performed.

Messages that Nix spit out during the above:

error (ignored): error: cannot unlink '/tmp/nix-build-rust-analyzer-unwrapped-2021-08-23.drv-0/source/target/x86_64-unknown-linux-gnu/release/deps': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-tree-sitter-0.20.0.drv-0/source/target/x86_64-unknown-linux-gnu/release/deps': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-seshat-node-2.3.0.drv-0/source/seshat-node/tmp/.cache/yarn/v6/npm-rxjs-6.6.7-90ac018acabf491bf65044235d5863c4dab804c9-integrity/node_modules/rxjs/src/operator': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-git-2.32.0.drv-0/git-2.32.0': Directory not empty
error: interrupted by the user

FWIW I've done the same on Nix 2.3 and haven't seen this error but occasionally had old build directories in /tmp. Nix 2.4. might just be more verbose about the situation.

Steps To Reproduce

1. Start a build using nix-build
2. Cancel the build with Ctrl-C
3. If cleanup failed, reapeat.
4. Eventually you should see something like above.

Expected behavior

I would expect Nix to cleanup all of the temporary build directories when a build is being aborted.

nix-env --version output

2.4.pre20210802_47e96bb

[oxalica]

Also run into this. In some rare cases, canceling a running build doesn't kill all build processes. nix-build exited but background process like gcc and make are still running. Is this the same issue? Running process can also lead to Directory not empty if they continue producing files.

[andir]

Also run into this. In some rare cases, canceling a running build doesn't kill all build processes. nix-build exited but background process like gcc and make are still running. Is this the same issue? Running process can also lead to Directory not empty if they continue producing files.

I guess so. I've seen that as well. I initially thought it was a thing regarding MacOS (as that was the first system I used Nix 2.4+ on as "daemon") but apparently that is also true for non-darwin?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-4-and-what-s-next/16257/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-4-and-what-s-next/16257/8

[stale[bot]]

I marked this as stale due to inactivity. → More info

[fogti]

Still important

[shd101wyy]

Yes I just encountered the same problem

[Madouura]

I haven't gotten this specific error, but I do see a lot of build dirs that I had to ctrl+c when testing. They just linger there forever unless I manually delete them.

[Madouura]

While this is a nix issue and IMO nix should be proactively cleaning it's build dirs in /tmp if they're not in use (locked), apparently boot.tmp.cleanOnBoot (at nixpkgs) has been set to false by default for a while now.

[tomberek]

Was it ever determined if this was a difference in behavior or only the presence of a warning bringing attention to it?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-rebuild-fails-on-a-weird-error-after-bumping-nixpkgs/50397/3

[bikallem]

I am facing a similar issue when trying to do :b pkgs.pkgsStatic.poppler in nix repl. I can see *.drv files in /tmp with owners set to root. In order to manually remove these, I have to do sudo rm -rf /tmp/*.drv.

[mitchty]

My solution for this for now was to simply add these dirs to tmpfiles rules for cleanup. Time chosen here to make sure I don't remove stuff that might be in use. Change it to whatever works for your needs.

  systemd = {
    tmpfiles.rules = [
      "R /tmp/nix-build-* - - - bmA:2h -"
    ];
  };

And then can kick it off straight away too if you are impatient:

sudo systemctl start systemd-tmpfiles-clean.timer

[mweinelt]

Age does not apply to R. You could try the following instead:

  systemd.services.prune-stale-nix-builds = {
    description = "Prune stale nix build roots";
    startAt = "hourly";
    unitConfig.Documentation = "https://github.com/NixOS/nix/issues/5207";
    serviceConfig = {
      ExecStart = lib.concatStringsSep " " [
        (lib.getExe pkgs.findutils)
        "/tmp"
        "-maxdepth 1"
        "-type d"
        "-iname \"nix-build-*\""
        "-mtime +1" # days
        "-exec rm -rf {} +"
      ];
    };
  };

[mweinelt]

Flagging as relevant for infra, as our linux builders build on a tmpfs. If we don't clear out leftover builds we'll eventually run out of space.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/clean-up-keep-failed-artifacts/62340/10

[Kek5chen]

~~Just received this issue as well:~~

root > rm -rf /tmp/*
root > nixos-rebuild switch
.....
~~ snip ~~
error: builder for '/nix/store/0fi8pcv50vg1zy1b1xnynr8jxy1pvcp6-source.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/q966dihhmifr1zzgk54wwrw5pj1j6ic2-ryujinx-1.2.76.drv' failed to build
error (ignored): error: cannot unlink '"/tmp/nix-build-heroic-unwrapped-2.16.1.drv-0/build/tmp.4LDLECpfQH/v3/files"': Directory not empty
error (ignored): error: cannot unlink '"/tmp/nix-build-element-desktop-1.11.96.drv-0/build/tmp.O1WB5fIAl7/.cache/yarn/v6/[email protected]/node_modules/@sentry/node/build"': Directory not empty
copying path '/nix/store/sd8hx5w51vrgd5mjxbc2pzgqbb5d2bks-gnome-bluetooth-47.1' from 'https://cache.nixos.org'...
error: 1 dependencies of derivation '/nix/store/2nhwxcspsimgwyp0pl3kfa0pzs5i4h02-system-path.drv' failed to build
copying path '/nix/store/hncq74xwajwyah4v7m45x5ndva6hapjd-mozc-2.30.5544.102' from 'https://cache.nixos.org'...
copying path '/nix/store/f72c4xvfdx82annffjhazmfymh16lmhw-xdg-desktop-portal-gnome-47.1' from 'https://cache.nixos.org'...
copying path '/nix/store/5vp08caq6fv81pr0zawfvbxy8skvajcx-gnome-online-accounts-3.52.3.1' from 'https://cache.nixos.org'...
copying path '/nix/store/sxidlhailnizkfbapf6qnxqjwjy4pfr6-gnome-clocks-47.0' from 'https://cache.nixos.org'...
copying path '/nix/store/2dy5vaxxhw0d6wxidasilcy15c03lhd9-nautilus-47.1' from 'https://cache.nixos.org'...
copying path '/nix/store/ch4558c89v2ggpdpmwibd46s97yfxkdv-tecla-47.0' from 'https://cache.nixos.org'...
copying path '/nix/store/7kzqfclcy4nfyjr1xylhbvk9b68m1ksl-webkitgtk-2.46.6+abi=6.0' from 'https://cache.nixos.org'...
copying path '/nix/store/pn5ciclgsjs8z5wqicpzcbls8vzizw59-rygel-0.44.1' from 'https://cache.nixos.org'...
error (ignored): error: cannot unlink '"/tmp/nix-build-nvidia-x11-565.77-6.6.85.drv-0/build/NVIDIA-Linux-x86_64-565.77/kernel"': Directory not empty
error: 1 dependencies of derivation '/nix/store/dxf9k26xjpkiq6kkmqkafsx6il4mxjch-nixos-system-nixos-24.11.716687.bdb91860de2f.drv' failed to build

I'm on a mix of 24.11 and unstable.

The original build error happens in ryujinx. Ignore this. It's just a side effect. I noticed the (ignored) only later on.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-05-28-nix-team-meeting-minutes-229/65205/1

[MelihDarcanxyz]

Having the same error. Additionally, like other people in this issue, compilation processes don't stop after quitting with Ctrl+C, had to manually track & kill them with btop.