doctest: Missing file: /nix/store/g1w99d0b9628lvwbyljm7hgrlnw5zc0n-ghc-9.6.2/lib/ghc-9.6.2/lib/settings

[runeksvendsen]

$ cabal install doctest --overwrite-policy=always && cabal exec -- doctest -isrc dump-decls-lib/src/
Warning: The package list for 'hackage.haskell.org' is 73 days old.
Run 'cabal update' to get the latest list of available packages.
Warning: The package list for 'hackage.haskell.org' is 73 days old.
Run 'cabal update' to get the latest list of available packages.
Resolving dependencies...
Symlinking 'doctest' to '/Users/rune/.cabal/bin/doctest'
Warning: The package list for 'hackage.haskell.org' is 73 days old.
Run 'cabal update' to get the latest list of available packages.
Resolving dependencies...
doctest: Missing file: /nix/store/g1w99d0b9628lvwbyljm7hgrlnw5zc0n-ghc-9.6.2/lib/ghc-9.6.2/lib/settings

I think the problem is that ghc-paths was compiled with one version of GHC in the PATH and now that version of GHC is no longer present where it was before. This happens because ghc-paths' path to ghc is fixed at compile-time of ghc-paths.

This is perhaps a duplicate of https://github.com/sol/doctest/issues/40. But that issue was closed, pointing to #42 for the reason, in which a comment mentions a workaround that I can't find. What's the workaround?

[sol]

Hmm, I think this message is originating from:

https://github.com/ghc/ghc/blob/c318be56f69eb87f1cbb219dd660f09498857ece/compiler/GHC/Settings/IO.hs#L50 I

I think the problem is that ghc-paths was compiled with one version of GHC in the PATH and now that version of GHC is no longer present where it was before.

I think the PATH is irrelevant for this. ghc-paths uses the ghc that is used to build ghc-paths itself. So as long as

1. you use the same version of ghc to build both ghc-paths and doctest, and
2. that version of ghc is still in place when you run doctest

things should work.

Should this be a nixpkgs issue instead? Have you tried to use doctest from nixpkgs instead? Did that work?

[runeksvendsen]

Hmm, I think this message is originating from:

https://github.com/ghc/ghc/blob/c318be56f69eb87f1cbb219dd660f09498857ece/compiler/GHC/Settings/IO.hs#L50 I

Yes. The cause is that a non-existent libdir is passed to a GHC library function, e.g. runGhc.

I think the problem is that ghc-paths was compiled with one version of GHC in the PATH and now that version of GHC is no longer present where it was before.

I think the PATH is irrelevant for this. ghc-paths uses the ghc that is used to build ghc-paths itself. So as long as

1. you use the same version of ghc to build both ghc-paths and doctest, and
2. that version of ghc is still in place when you run doctest

things should work.

Yes, I may be mistaken regarding which GHC is used by ghc-paths (the one on the PATH or the one used to build itself). But the problem is that I don't know how to rebuild ghc-paths now that the GHC that was once used to build it is gone. If I could somehow force a rebuild of ghc-paths, with the GHC version that I'm using now, the issue would be solved. But I don't know how to do that.

If this isn't even possible then I think this is a bug in the design of ghc-paths, and hence in doctest, that should at least have a workaround. Otherwise I'm stuck here.

Should this be a nixpkgs issue instead?

All nixpkgs is doing is providing a GHC version that I have previously used to build both ghc-paths and doctest. Now that GHC version is no longer present in the location it was before, and this isn't handled properly by ghc-paths. I don't see how nixpkgs can be blamed for this.

Have you tried to use doctest from nixpkgs instead? Did that work?

Yes, I tried it and it gives me an error when running doctest:

[nix-shell:~/code/dump-decls]$ doctest -isrc dump-decls-lib/src/

dump-decls-lib/src/Compat/Aeson.hs:11:5: error:
     error: function-like macro 'MIN_VERSION_aeson' is not defined
   |
11 | #if MIN_VERSION_aeson(2,0,0)
   |     ^
#if MIN_VERSION_aeson(2,0,0)
    ^
1 error generated.

dump-decls-lib/src/Compat/Aeson.hs:1:1: error:
    `cc' failed in phase `C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

[sol]

So your ghc-paths is not provided by nixpkgs, right? In that case you can try to rm -rf ~/.local/state/cabal/store. Please try this first and then report back.

I have more to say, but switching device, so give me a second.

[sol]

I have more to say, but switching device, so give me a second.

So continuing here, firstly, what version of GHC are you using? Only GHC 9.8 and later include ABI hashes in the package IDs of boot libraries. If you are using anything older than that then I think you are much more likely to run into issues.

[sol]

But even with a recent version of GHC, what I imagine could happen is that some nixpkgs dependency of GHC changes, resulting in a different store path, while the GHC ABI hash does not change.

If that happens, then I think you're in trouble. rm -rf ~/.local/state/cabal/store should help as a workaround.

I'm not really into Nix and I am not sure how others deal with this. I think @soenkehahn has a lot of relevant experience here. Maybe he is willing to chime in. Maybe garn addresses this somehow, not sure?

Other than that, this is what I can think of:

1. Cabal could include the canonical path to GHC in store hashes (=> Open a Cabal issue)
2. Nixpkg could try to ensure that the ABI hash changes when the store path changes. Maybe they already do something here, which would mean that my hypothesis from above is wrong. This would affect every Nix user, so I would be kind of surprised if there is no solution available. Anyway in this case this would mean opening a nixpkgs issue (if nixpkgs wants to directly influence the ABI hash, then possibly also a GHC issue, not sure).
3. Technically it would be possible to extend doctest so that the user can provide the path to GHC, e.g. through an environment variable. However, as doctest only works with the ghc version it was compiled against, I'm not sure if this is the right approach.

[sol]

Yes, I tried it and it gives me an error when running doctest:

[nix-shell:~/code/dump-decls]$ doctest -isrc dump-decls-lib/src/

dump-decls-lib/src/Compat/Aeson.hs:11:5: error:
     error: function-like macro 'MIN_VERSION_aeson' is not defined
   |
11 | #if MIN_VERSION_aeson(2,0,0)
   |     ^
#if MIN_VERSION_aeson(2,0,0)
    ^
1 error generated.

dump-decls-lib/src/Compat/Aeson.hs:1:1: error:
    `cc' failed in phase `C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

As for running doctest, please make sure that it works with a minimal example that does not use CPP.

If you then wanna run it for a cabal package you have these options:

• Manually provide all required options for doctest
• Use cabal repl
• Use cabal doctest

Please read the README for how to do this.

[runeksvendsen]

So your ghc-paths is not provided by nixpkgs, right?

Correct. Only GHC and cabal-install are provided by nixpkgs; everything else I do by running cabal-related CLI commands.

In that case you can try to rm -rf ~/.local/state/cabal/store. Please try this first and then report back.

I have no directory ~/.local/state/cabal/store, but I do have ~/.cabal/store. Temporarily renaming ~/.cabal/store/ghc-9.6.2 to something else forces a rebuild of all dependencies when I run cabal install doctest --overwrite-policy=always && cabal exec -- doctest -isrc dump-decls-lib/src/ and therefore gets rid of the error. I guess this is the workaround I was looking for, although it would be preferable if I could wipe only ghc-paths and not all other packages for the same GHC version.

So continuing here, firstly, what version of GHC are you using? Only GHC 9.8 and later include ABI hashes in the package IDs of boot libraries. If you are using anything older than that then I think you are much more likely to run into issues.

I'm using GHC 9.6.2

But even with a recent version of GHC, what I imagine could happen is that some nixpkgs dependency of GHC changes, resulting in a different store path, while the GHC ABI hash does not change.

I'm not sure what this means, but the issue here is very clear to me: when ghc-paths is compiled, the path to whatever GHC version was used compile it is embedded into the resulting object code. If, later, this path no longer exists then everything compiled against this object code will fail at runtime.

[sol]

What I'm saying is, if you can, upgrade to a more recent version of GHC first (at least 9.8.*). If you don't do that, you have to be prepared to run into arbitrary ABI issues if you rebuild ghc-9.6.2 with a new version of nixpkgs. (see also https://github.com/haskell/cabal/issues/8114)

If you can't upgrade then I recommend to remove your cabal store after every rebuild of ghc. This will both help with ABI issues and with your ghc-paths issue.

If you can upgrade, you should not run into ABI issues and the ghc-paths issue should also be less likely to occur. But I kind of assume that it could still happen. How to address this properly, I I don't know. I am inclined to say that this is a nixpkgs issue and they need to come up with a proposal.

[sol]

For completeness:

With GHC 9.8+

• Whenever the path to ghc in the nix store changes, but the Project Unit Id (look at ghc --info) does does not change then ghc-paths will be broken.

Edit: Just to be clear, all packages, including ghc-paths will be rebuilt when the Project Unit Id changes. That's why you will only run into issues with ghc-paths when the Project Unit Id doesn't change.

With GHC 9.6.*

• Whenever the path to ghc in the nix store changes then ghc-paths will be broken + you may run into arbitrary ABI issues.