haskell
Too many duplicates in hsc command line arguments, causing build failures due to GCC argument limits
I couldn't find an open bug report for this. Feel free to close this as a duplicate if it already exists!
Describe the bug
When cabal invokes hsc, it builds a list of arguments that among other things, include linker arguments for linking against all of the package's (transitive) dependencies' C dependencies:
https://github.com/haskell/cabal/blob/e7657ad319b9a5b50718262488d17aad915aad90/Cabal/src/Distribution/Simple/PreProcess.hs#L609-L622
Because a lot of packages tend to depend on the same set of libraries, this ends up adding many duplicates of the same argument. Normally this doesn't cause any real problems because gcc and the linker will ignore these duplicates, but GCC also has an argument length limit. If you have enough dependencies then you will eventually trip this limit (see https://github.com/NixOS/nixpkgs/issues/41340, though this hsc issue is a separate issue).
In a codebase I'm working on right now, hsc adds hundreds of the same arguments, causing it to trip this limit. This is an xcerpt from the the gcc command line, split on spaces, sorted by the number of occurrences:
...
5 -L/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
5 -L/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
5 -L/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
5 -Wl,-R,/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
5 -Wl,-R,/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
5 -Wl,-R,/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
7 -L/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
7 -Wl,-R,/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
269 -L/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
269 -Wl,-R,/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
270 -L/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
270 -L/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
270 -L/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
270 -Wl,-R,/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
270 -Wl,-R,/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
270 -Wl,-R,/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
In this case, the cabal build command terminates with the following error:
Build profile: -w ghc-9.6.5 -O1
In order, the following will be built (use -v for more details):
- libname-0 (lib) (first run)
Preprocessing library for libname-0..
linking /repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/Buffers_hsc_make.o failed (exit code 1)
rsp file was: "/repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/hsc2hscall2431973-2.rsp"
command was: /nix/store/mpm3i0sbqc9svfch6a17179fs64dz2kv-gcc-wrapper-13.3.0/bin/cc <260 KiB worth of command line arguments>
error: gcc: fatal error: cannot execute ‘/nix/store/zc0nsv23pakbafngjy32kvhfzb16as43-gcc-13.3.0/libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/collect2’: execv: Argument list too long
compilation terminated.
Error: .cabal-wrapped_: Failed to build libname-0.
I think simply deduplicating these arguments should fix the issue.
To Reproduce I sadly cannot provide a reproducer because the project in question is not public, but this will happen on any project with a sufficiently large amount of dependencies. The problem is exacerbated by using Nix, since the paths to the library directories end up being much longer that way.
Expected behavior
c2hs/hsc should compile the module as expected.
System information
- Ubuntu 22.04
- nixpkgs pin
1e3deb3d8a86a870d925760db1a5adecc64d329d - GHC 9.6.5 provided through Nix
- Cabal 3.10.3.0 provided through Nix
Additional information I can provide a patch for this if needed. There may also be other command line invocations with similar problems.
