`nix-collect-garbage -d` does not clean up user profiles in XDG directories when run as root

[K900]

Describe the bug

❯ ls .local/state/nix/profiles 
home-manager@  home-manager-14-link@  home-manager-15-link@  home-manager-16-link@  profile@  profile-12-link@  profile-13-link@  profile-14-link@

❯ sudo nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/per-user/root/profile
removing old generations of profile /nix/var/nix/profiles/per-user/root/home-manager
removing profile version 257
removing profile version 258
removing old generations of profile /nix/var/nix/profiles/per-user/k900/profile
removing old generations of profile /nix/var/nix/profiles/system
removing profile version 502
removing profile version 503
removing old generations of profile /nix/var/nix/profiles/default
removing old generations of profile /nix/var/nix/profiles/per-user/root/profile
removing old generations of profile /nix/var/nix/profiles/per-user/root/home-manager
finding garbage collector roots...
<snip>

❯ ls .local/state/nix/profiles
home-manager@  home-manager-14-link@  home-manager-15-link@  home-manager-16-link@  profile@  profile-12-link@  profile-13-link@  profile-14-link@

❯ sudo nix-collect-garbage -d --option use-xdg-base-directories true
removing old generations of profile /nix/var/nix/profiles/per-user/root/profile
removing old generations of profile /nix/var/nix/profiles/per-user/root/home-manager
removing old generations of profile /nix/var/nix/profiles/per-user/k900/profile
removing old generations of profile /nix/var/nix/profiles/system
removing old generations of profile /nix/var/nix/profiles/default
removing old generations of profile /nix/var/nix/profiles/per-user/root/profile
removing old generations of profile /nix/var/nix/profiles/per-user/root/home-manager
finding garbage collector roots...
<snip>

❯ ls .local/state/nix/profiles
home-manager@  home-manager-14-link@  home-manager-15-link@  home-manager-16-link@  profile@  profile-12-link@  profile-13-link@  profile-14-link@

❯ nix-collect-garbage -d     
removing old generations of profile /home/k900/.local/state/nix/profiles/home-manager
removing profile version 14
removing profile version 15
removing old generations of profile /home/k900/.local/state/nix/profiles/profile
removing profile version 12
removing profile version 13
removing old generations of profile /nix/var/nix/profiles/per-user/k900/profile
finding garbage collector roots...
<snip>

Expected behavior

home-manager generations are cleaned up on the first sudo nix-collect-garbage -d invocation.

nix-env --version output

nix-collect-garbage (Nix) 2.16.1

Additional context

I'm not sure when this started happening, but I feel like it might have something to do with HM switching to XDG paths?

Priorities

Add :+1: to issues you find important.

[vamega]

In your example, are those commands being run as root, or as a regular user?

I'm only seeing the root home-manager profiles persisting.

[K900]

The commands prefixed with sudo are running as root, the commands not prefixed with sudo are not running as root.

[vamega]

Ah of course. Early morning and i wasn't thinking straight.

I was able to clear out those symlinks (which in my case only exist for root) by running

nix-env --profile /root/.local/state/nix/profiles/home-manager --delete-generations +5

I noticed this morning when building a derivation failed due to a lack of disk space.

[Ericson2314]

This is intentional. We didn't want the command snooping around in home directories of other users. I think in the docs or release notes it says one should use sudo -u to go impersonate users and clean each one if they wanted to do the old behavior.

But if you think this is a bad decision, feel free to offer alternatives. e.g. the indirect roots in the store dir can indicate which users are worth sudo -u-ing.

[K900]

Maybe just add an --all-users flag to get the old behavior back? It would still be opt-in, but easier than running the command twice.

[Ericson2314]

Just saying "let there be a flag" is the easy part. Please think through the details a bit more.

Before, we could see all the (non custom location) profiles in /nix/var/nix/profiles/per-user. Now what are we supposed to do, look in every user's home directory? That is why I mentioned trying to look at indirect GC roots. What do we do if the home directory is not there (e.g. remote LDAP users with certain configs)? There are probably other corner cases too.

[K900]

Oh wait, I see the problem - the new profiles aren't linked in profiles/per-user. Then yeah, indirect GC roots are probably the solution, though it would need some extra plumbing on the HM side.

[vamega]

Where are these release notes/documentation.

I was looking at https://nixos.org/manual/nix/stable/introduction.html and couldn't really find anything about this. I also looked at the nixos release notes.

If it's missing it might be worth adding this somewhere. Didn't cause me a lot of trouble, but would be nice to not have a more official place where this information is surfaced than this ticket.

Thanks for the help.

[viperML]

This issue should be top priority

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/home-manager-and-garbage-collection/41715/2

[dcarosone]

I suspect there are a number of things that can help here. One might just be the search path confusion I mention in the discourse post above; we'll see.

Another is that, basically, profiles can be anywhere, and they can be used for generations of a devshell in a workspace or other things. So it's not just home-manager, although again the coupling to system generations is relevant for that case. And finding profiles is hard. Perhaps they're not even accessible at the time gc is run, because the user's homedir is encrypted with per-user keys via pam_zfs or one of several other options.

So I think part of the issue is to have some cleanup in the profile generation mechanism itself: the time to expire old generation links is when making a new generation that can replace it, based on a similar policy to the gc batch job (at most N generations, at most M days old, etc). Those links can be removed at time of issue, and the store gc batch can just work with the roots that are left at the end of the day.

(/cc @samueldr because rumour says this is RTYI)

[0xDubdub]

For anyone else surprised to find a mountain of uncollected garbage despite using nix.gc, I’ve set up a systemd timer by adjusting my configuration as follows:

  # Automatic garbage collection (system-wide profiles)
  nix.gc = {
    automatic = true;
    dates = "daily";
    options = "--delete-older-than 7d";
    randomizedDelaySec = "5min";
  };

  # Automatic garbage collection (user profiles)
  systemd.user.services."nix-gc" = {
    description = "Garbage collection for user profiles";
    script = "/run/current-system/sw/bin/nix-collect-garbage --delete-older-than 7d";
    startAt = "daily";
  };

[maralorn]

I am thinking of writing the following bash script:

1. run nix-store --gc --print-roots
2. Match for paths of the form <path>/<name>-<number>-link
3. Run nix-env --delete-generations -p <path>/<name> for every of them.

Is there a reason why this would be wrong? And why this couldn’t be the default behavior of nix-collect-garbage -d?

[dcarosone]

Aside: here's what I currently use to review what needs cleaning up from time to time: https://discourse.nixos.org/t/show-closure-size-of-garbage-collector-roots/29118/9?u=uep

Match for paths of the form /--link

yes, noting that can also have -'s in it, e.g. "home-manager", obviously

Is there a reason why this would be wrong?

It shouldn't delete the current generation, even if that isn't the most recent due to a rollback. It might delete newer ones in that case, and just in general different profiles might want different policies. A remote user who logs in only occasionally may want a different history period for their h-m profile. I don't know what the thinking was, but perhaps this was part of the reason it's no longer that default.

[mivalov]

For those using Home Manager and want to automatically run the garbage collector for the user profile(s), you could use the following snippet as part of the HM configs:

nix.gc = {
  automatic = true;
  # Up to and including HM release 25.05
  frequency = "weekly";
  # As of HM release 25.11 or current unstable branch: "frequency" -> "dates"
  dates = "weekly";
  options = "--delete-older-than 14d";
};

More information about the options can be found on their documentation page: https://nix-community.github.io/home-manager/options.xhtml#opt-nix.gc.automatic

[dcarosone]

Neat!

One minor thought: it might be preferable to have separation between expiring the profile links as gc roots and garbage-collecting the store for all the now-garbage paths, since (due to the root cause of this issue) there seems to now be a need to run per-user as well as system instances of the first. The second actual store gc is a more io-intensive operation that is likely best performed once for the aggregated result, or at least at a system-defined time and interval. It might make things less impactful: on a server where there are lots of users (to avoid repeated execution and scanning), and on a single user workstation (to minimise the thundering herd of persistent jobs that get run after a login when the machine has been off for a while).

They're accessible as separate operations, at least in the new cli, via nix profile wipe-history and nix store gc. The latter is also available as nix-store --gc (and nix-collect-garbage without additional profile-expiry options). I'm not sure of a way to do just the profile link expiry using the old/stable cli, but haven't looked very hard.

But of course, some coordination is required, and there might not even be a system-level service, it might be a single user h-m and nix on some other platform, etc etc. This is certainly helpful, especially for those stand-alone cases.