Use response files for `ghc` invocations

[Gabriella439]

Before this change, cabal could fail with the following error message when building very large Haskell packages:

ghc: createProcess: posix_spawnp: resource exhausted (Argument list too long)

This is because when the number of modules or dependencies grows large enough, then the ghc command line can potentially exceed the ARG_MAX command line length limit.

However, ghc supports response files in order to work around these sorts of command line length limitations, so this change enables the use of those response files.

Note that this requires taking a special precaution to not pass RTS options to the response file because there's no way that ghc can support RTS options via the response file. The reason why is because the Haskell runtime processes these options (not ghc), so if you store the RTS options in the response file then ghc's command line parser won't know what to do with them.

This means that ghc commands can still potentially fail if the RTS options get long enough, but this is less likely to occur in practice since RTS options tend to be significantly smaller than non-RTS options.

This also requires skipping the response file if the first argument is --interactive. See the corresponding code comment which explains why in more detail.

Please read Github PR Conventions and then fill in one of these two templates.

---

Template Α: This PR modifies cabal behaviour

Include the following checklist in your PR:

• [x] Patches conform to the coding conventions.
• [ ] Any changes that could be relevant to users have been recorded in the changelog.
• [ ] The documentation has been updated, if necessary.
• [ ] Manual QA notes have been included.

Bonus points for added automated tests!

---

Template Β: This PR does not modify cabal behaviour (documentation, tests, refactoring, etc.)

Include the following checklist in your PR:

• [ ] Patches conform to the coding conventions.

Depends-on: #10292

[Gabriella439]

Note that this has been tested internally at Mercury not just for cabal commands but also downstream tools (like ghcid and haskell-language-server), which is how we stumbled upon the hie-bios issue.

We ran into this issue when building our internal Haskell monolith (which has ~7000 Haskell modules), and we verified that this change fixes the problem without breaking anything else.

[Kleidukos]

@Gabriella439 There are still some failures: https://github.com/haskell/cabal/actions/runs/6643530708/job/18050770478?pr=9367#step:16:14024

[Gabriella439]

I'm trying to reproduce the problem locally, but when I run ./validate.sh -s build && ./validate.sh -s lib-suite I get a whole bunch of tests being skipped with "SKIP no cabal-install" and I'm not sure how to fix that

[Mikolaj]

Would any of these help?

https://github.com/haskell/cabal/blob/master/CONTRIBUTING.md#running-tests https://github.com/haskell/cabal/blob/master/cabal-testsuite/README.md

[9999years]

The requested support for preserving the temp files ended up being much more complicated to implement than I thought it would be, and I just wanted to check with the maintainers to make sure I'm on the right track.

I've put the flag in SetupCommonOptions because those are already passed to the build functions that need them, and I've attempted to thread it through the conversion functions in D.C.ProjectConfig.Legacy, but I keep getting test failures from the value getting dropped somewhere in the round-trip conversion; this is with cabal test cabal-install:unit-tests --test-options="-p ProjectConfig":

projectConfigKeepTempFiles = -NoFlag +Flag False,

Am I missing something? Is there a simpler way of doing this?

[9999years]

Ready for review, but we should probably merge #10292 first for the --keep-temp-files changes.

[geekosaur]

Added a dependency so Mergify will hold off until it's merged.

[9999years]

Ready for review! #10292 is merged and I've added a release note.

[Mikolaj]

@mpickering: I'm guessing your comment is addressed? Would you like have a second look?

[mergify[bot]]

This pull request has been removed from the queue for the following reason: pull request branch update failed.

The pull request can't be updated.

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

[geekosaur]

@Gabriella439, since you made this PR from your work account, you must rebase it manually and use the merge+no rebase label because neither Mergify nor we are allowed to update it (no access to your employer's GitHub organization).

[mergify[bot]]

This pull request has been removed from the queue for the following reason: pull request dequeued.

Pull request #9367 has been dequeued. Pull request automatically merged by a merge action.

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue