devShell inconsistencies and strange behaviour

[chessai]

I have a flake that looks very similar to https://github.com/Liqwid-Labs/plutus-extra/blob/master/flake.nix.

When I check out that repo (Liqwid-Labs/plutus-extra) and enter a dev shell with nix develop, the environment is fine and I can build everything with cabal. But I noticed something strange: All of the packages added from cabal.project are specified in additional. In fact, when removing additional and entering a shell with nix develop, the specified packages disappear from the package registry (ghc-pkg list). This is really strange; shouldn't haskell.nix auto-populate the package registry with these packages, since they are needed to build the project? I don't see this additional being used in this way in other haskell.nix projects - just in this singular one I could find that is using flakes (most haskell.nix projects don't seem to fully use flakes yet). Perhaps flake compatibility is an issue, or devShell is doing something different than what nix-shell does that breaks this?

Secondly, on a closed source project, specifying the things in additional isn't enough, because nix develop fails, complaining about cross compilation, even though I'm not cross-compiling. Perhaps the issue is I'm using pkgsCross.musl64?

I was about to paste the error I got - but when I went to go reproduce the error, the following sequence of events happened and I've lost the error to scrollback:

1. I ran nix develop -L with a shell = { ...some stuff...; additional = ps: [ ps.Win32-network ]; }, got an error about cross compilation, we'll call this error A
2. Commented out entirety of shell = { ...stuff... }, ran nix develop -L, got a different error (expected/understandable), call this error B
3. uncommented shell = { ...stuff... }, ran nix develop -L, got error B again, not error A, despite the nix expression returning the state of (1)

So now, I can't reproduce error A. But I'll paste error B since it's what's happening now, I just wanted to document the strangeness. Here is error B:

postgresql-libpq-lib-postgresql-libpq-x86_64-unknown-linux-musl> Setup: The program 'pg_config' is required but it could not be found.
error: builder for '/nix/store/4d2h50zk734qlkmi7n4yv4vp1z1z3ann-postgresql-libpq-lib-postgresql-libpq-x86_64-unknown-linux-musl-0.9.4.3.drv' failed with exit code 1;
       last 10 log lines:
       > source root is postgresql-libpq-0.9.4.3
       > setting SOURCE_DATE_EPOCH to timestamp 1000000000 of file postgresql-libpq-0.9.4.3/test/Smoke.hs
       > patching sources
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > Configure flags:
       > --prefix=/nix/store/nzbmi8s83p50rvl0nx9mn0mbb9n5n57s-postgresql-libpq-lib-postgresql-libpq-x86_64-unknown-linux-musl-0.9.4.3 lib:postgresql-libpq --extra-include-dirs=/nix/store/vpyznj3q6yb6sk9r9rnz0k4mzr9a88na-postgresql-x86_64-unknown-linux-musl-13.5/include --extra-lib-dirs=/nix/store/86hq4c1hb4sqpygfyzla4dhyrn6lhjjr-postgresql-x86_64-unknown-linux-musl-13.5-lib/lib --package-db=clear --package-db=/nix/store/mrdqx60xiyclqvg6q0ji4g4dx8cn50i1-x86_64-unknown-linux-musl-postgresql-libpq-lib-postgresql-libpq-0.9.4.3-config/lib/x86_64-unknown-linux-musl-ghc-8.10.7/package.conf.d --flags=-use-pkg-config --exact-configuration --dependency=rts=rts --dependency=ghc-heap=ghc-heap-8.10.7 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=integer-gmp=integer-gmp-1.0.3.0 --dependency=base=base-4.14.3.0 --dependency=deepseq=deepseq-1.4.4.0 --dependency=array=array-0.5.4.0 --dependency=ghc-boot-th=ghc-boot-th-8.10.7 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.16.0.0 --dependency=ghc-boot=ghc-boot-8.10.7 --dependency=ghc=ghc-8.10.7 --dependency=Cabal=Cabal-3.2.1.0 --dependency=array=array-0.5.4.0 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.5.1 --dependency=directory=directory-1.3.6.0 --dependency=filepath=filepath-1.4.2.1 --dependency=ghc-boot=ghc-boot-8.10.7 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=hpc=hpc-0.6.1.0 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.14.0 --dependency=process=process-1.6.13.2 --dependency=text=text-1.2.4.1 --dependency=time=time-1.9.3 --dependency=transformers=transformers-0.5.6.2 --dependency=unix=unix-2.7.2.2 --dependency=xhtml=xhtml-3000.2.2.1 --with-ghc=x86_64-unknown-linux-musl-ghc --with-ghc-pkg=x86_64-unknown-linux-musl-ghc-pkg --with-hsc2hs=x86_64-unknown-linux-musl-hsc2hs --with-gcc=x86_64-unknown-linux-musl-cc --with-ld=x86_64-unknown-linux-musl-ld.gold --ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ar=x86_64-unknown-linux-musl-ar --with-strip=x86_64-unknown-linux-musl-strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-profiling --enable-static --enable-shared --disable-coverage --enable-library-for-ghci --enable-split-sections
       > Configuring library for postgresql-libpq-0.9.4.3..
       > Setup: The program 'pg_config' is required but it could not be found.
       >
       For full logs, run 'nix log /nix/store/4d2h50zk734qlkmi7n4yv4vp1z1z3ann-postgresql-libpq-lib-postgresql-libpq-x86_64-unknown-linux-musl-0.9.4.3.drv'.
error (ignored): error: cannot unlink '/tmp/nix-build-elfutils-x86_64-unknown-linux-musl-0.186.drv-0/elfutils-0.186/tests': Directory not empty
error: 1 dependencies of derivation '/nix/store/a7gkyj6gi14pk0aw0lx04sr653lpanxh-ghc-shell-for-packages-x86_64-unknown-linux-musl-env.drv' failed to build

Some questions:

1. Why is it building anything at all? I've already run nix build .#flake.x86_64-linux.packages."my_package" - every dependency necessary to build the project should already exist in the nix store.
2. Why is additional necessary?

I'll work on a minimal reproduction and post when I'm done (if I'm successful).

[chessai]

Minimal repro at https://github.com/chessai/haskell-nix-repro/commit/df3a3d1b6771d341835bd50affcfdb017bfb2b62

Curiously, I'm getting error A, not error B, and commenting out shell = { ...stuff... } causes nix develop to succeed.

[chessai]

Curiously, changing pkgsCross.musl64 to pkgs resolves the issue. I still need static linking for deploys, and I wanted to just use the same package set for local dev/deploys for convenience, unfortunately that isn't possible.

the fix is to change (using rough types) project :: System -> Project to project :: StaticOrNot -> System -> Project, using pkgs if not static, and pkgsCross.musl64 if static.

Another unfortunate side effect of this is an immense amount of duplication in our cache.

[chessai]

Actually, I spoke too soon. When I add the packages back to additional (which is still needed for some reason), nix develop fails, failing to build one of the packages. Here is some nix repl high-level debugging:

$ nix repl
> :lf .
> :b projectStatic.x86_64-linux.hsPkgs.playground-common.components.library
<<<fails>>>
> :b projectStatic.x86_64-linux.pkgs.haskell.packages.ghc8107.playground-common
<<< `playground-common` doesn't exist>>>
> :b flakeStatic.x86_64-linux.packages."transaction-writer-utils:exe:transaction-writer"
<<<builds just fine>>>

playground-common is a transitive dependency of transaction-writer. So, somehow, nix develop is using a different package set than nix build when it comes to the dependencies needed to build the project. And playground-common exists in packagesStatic.x86_64-linux.hsPkgs, but it's clearly a different version, and it doesn't exist at all in packagesStatic.x86_64-linux.pkgs.haskell.packages.ghc8107. But this leaves me wondering, where is the playground-common (and every other package specified in the cabal.project!) my build is actually using?

[chessai]

Okay, an actual fix was to abandon additional and use packages = ps: [ ps.my_package ]. Still not sure where the discrepancies stem from, but at least I have something working now.

[angerman]

Maybe @hamishmack has a clue?