shellFor: creates duplicate `buildInputs` on large projects, exceeding argument length limits

[JackKelly-Bellroy]

I work on a medium-sized haskell monorepo (~40KLOC spread over 30 packages), and discovered that adding a dependency that already existed caused nix-shell and nix develop -f . to begin failing with "argument list too long" when trying to execute bash.

I checked the (list) length of buildInputs on the derivation returned by shellFor, and it had 1699 entries, many of them repeated. As a workaround I have been able to bring the list back down to its 134 unique entries:

shell.overrideAttrs (oldAttrs: {
  buildInputs = pkgs.lib.unique oldAttrs.buildInputs;
})

However lib.unique is an O(n^2) operation. While it doesn't seem to be adversely affecting our evaluation time, perhaps there's a better long-term solution.

[JackKelly-Bellroy]

Suggestions from Nix upstream:

• Use __structuredAttrs = true;, as Nix >= 2.4 can handle that (and possibly Nix 2.3 too).
• Use listToAttrs to dedup the list in a faster way:

dedupByStringKey = keyFun: list:
  lib.attrValues (lib.listToAttrs (map (item: { name = keyFun item; value = item; }) list))

[peterbecich]

Attempt at using __structuredAttrs: https://github.com/peterbecich/haskell.nix/commit/dc4a3eb9d9ac52564e647a821be404c77307f368

The test suite fails:

% ./test/tests.sh ghc8107
...
default-builder.sh: line 1: /setup: No such file or directory

Could this be caused by an undefined $out? https://github.com/NixOS/nix/issues/3525#issuecomment-617746049

[avieth]

Consider closing this since #1613 has been merged?

[michaelpj]

Yes, thanks!