Tools barf on invalid (empty name) store paths, without telling me where they are referred to

[cumber]

Describe the bug

Certain nix tools that deal with my whole store tell me this:

❯ nix store verify --all
error: store path '4w6hv3912fjvanj4c9n83f3ahpql6ilj-' has an empty name

❯ nix run 'nixpkgs#nix-du'
Reading dependency graph from store... error: store path '4w6hv3912fjvanj4c9n83f3ahpql6ilj-' has an empty name
Could not read from store

This store path is not actually present on my system:

❯ ls /nix/store/4w6hv3912fjvanj4c9n83f3ahpql6ilj-
ls: cannot access '/nix/store/4w6hv3912fjvanj4c9n83f3ahpql6ilj-': No such file or directory

As a side note, this is in fact the same store path that I reported prevented me from running a garbage collection in #5015, which resulted from me years ago copying a Nixos module recommended on the wiki. At the time empty names were not forbidden, but I have long since stopped using this module and garbage collected the system generations that used it. Whatever is referring to it now is no longer preventing garbage collection; I don't know if this is because the garbage collector now tolerates cleaning up things that refer to invalid store paths, or because the garbage collector happens not to scan anything that refers to it.

Unfortunately, attempting to query for references to this store path (so that I can manually remove the things that refer to it) doesn't work; the query tool errors out on being asked to contemplate an invalid store path:

❯ nix-store -q --referrers /nix/store/4w6hv3912fjvanj4c9n83f3ahpql6ilj-
error: store path '4w6hv3912fjvanj4c9n83f3ahpql6ilj-' has an empty name

Steps To Reproduce

The query part of this is trivial to reproduce:

1. nix-store -q --referrers /nix/store/4w6hv3912fjvanj4c9n83f3ahpql6ilj-

But my real problem is that somewhere in my store is a reference to such a path (not the path itself), and that is difficult to reproduce now. I have this problem because I once-upon-a-time built something like builtins.toFile "" "contents of file with bad name" (as I mentioned, this was from this Nixos module, which I found suggested at https://nixos.wiki/wiki/Scanners#Network_scanning). The oldest nix in unstable nixpkgs now is nix_2_3, which won't allow such a derivation to be evaluated, let alone built:

nix-repl> builtins.toFile "" "contents of file with bad name"
error: error: store path '7ln9knqfd6gwf9jnrlb1pxabag99sy3i-' has an empty name

I believe nix 2.2 allowed such a build, however, if you get nix from an older nixpkgs. Having obtained such a nix, building builtins.toFile "" "contents of file with bad name" should reproduce all of the other symptoms I describe here.

Expected behavior

For nix store verify in particular, I would expect it not to fail on this error. Given its whole purpose is to "verify the integrity of store paths", it should at minimum report some diagnostics about which store paths it is processing when it finds the reference to an invalid store path; preferably it should also be able to recover and continue scanning for more problems.

I'm not sure whether it's practical for this error to be more informative in general, whatever context it occurs in; I imagine the point in the code that raises the error may not have easy access to knowledge of where the bad path was found.

For nix-store -q it would be awfully handy if passing it an invalid store path was only a warning, not a fatal error. But given that nix no longer lets you create such paths, and that the only reason I want that is to be able to find where the invalid path is referenced in order to remove the reference, it may not be worth anyone's trouble to make nix-store -q tolerate such an error, provided that there is some tool (I would expect nix store verify) that can find it.

nix-env --version output

nix-env (Nix) 2.6.0

Additional context

❯ nixos-version
22.05.20220207.60c52a7 (Quokka)

[itsfarseen]

How I solved this

I catd the file and it's content was same as one of the kernel patches I'm applying.
I was using fetchUrl to fetch this patch. I changed this to use a local copy. Either this, or the fact that I switched to nixos-unstable and then ran nixos-rebuild switch --upgrade, stopped the file from being written.

After doing the above, I restarted the system so that the current system in no way refers to this derivation.

Then I changed the version of nix by using nix.package = pkgs.nix_2_3 in ./etc/nixos/configuration.nix.
I was using pkgs.nixFlakes. Since I was enabling flake support, I also had to comment out the extraOptions that I was using to enable flakes support.
(My configuration.nix was still in non-flakes format)

The reason we have to downgrade is that nix tools choking on empty package names happen from v2.4+.

Once we have nix 2.3, use this to remove old boot generations:

nix-env -p /nix/var/nix/profiles/system --delete-generations old
nix-collect-garbage -d

Use this to remove boot entries for old generations:

# Get the name of current generation from here
nix-env -p /nix/var/nix/profiles/system --list-generations

# Replace <current-generation-name> with what you got from previous step.
# For example: 96
sudo bash -c "cd /boot/loader/entries; ls | grep -v <current-generation-name> | xargs rm"

Courtesy of this script: https://gist.github.com/xeppaka/f6126eebe030a000aa14ed63cc6e8496

Now reboot again, and run nix-collect-garbage -d.

Double check the file is gone.
Then revert the change to nix.package to get the original nix version back.

[cstich]

Once you downgrade to nix_2_3 you can also use nix-store --verify --repair --check-contents to fix the broken entries in the nix sqlite database. That fixed the issue for me.

[cumber]

@cstich Thank you, that's awesome! That has finally fixed this issue for me.

@itsfarseen You can just use e.g. open a shell with nix shell 'nixpkgs#nix_2_3 to run older versions of nix that are still in nixpkgs, no need to downgrade your system-wide nix with nix.package = pkgs.nix_2_3. Your procedure wasn't helping me because the file itself was long since gone from my system, but something was obviously still containing a reference to the path. I guess that was the sqlite database?

[cstich]

@cumber I also had the same issue. The offending derivation was long gone from my system. I couldn't find anything that referenced it but there was still a broken reference to it in the database. I am glad it works now for you as well. This took me the better part of a Sunday to figure out.

[cumber]

Weirdly enough, I happened to run a nix store verify --all again today, and the same error: store path '4w6hv3912fjvanj4c9n83f3ahpql6ilj-' has an empty name was back again. The same procedure seems to have gotten rid of the reference again:

❯ nix shell 'nixpkgs#nix_2_3'

❯ sudo nix-store --verify --repair --check-contents
warning: unknown setting 'experimental-features'
reading the Nix store...
checking path existence...
path '/nix/store/4w6hv3912fjvanj4c9n83f3ahpql6ilj-' disappeared, removing from database...
checking hashes...

I'm a little concerned that this reference to a bad path is somehow being resurrected, years after I have removed it from my system!

[tomberek]

Likely needs work in the repair subsystem to allow fixing incorrect metadata.