Cross compilation support

Open andreabedini opened this issue 4 months ago • 1 comments

Hello,

this PR adds support for cross-compilation to cabal-install.

In this work cross-compilation is viewed exclusively as a cabal-install feature, and it is implemented without any substantial change to Cabal. I introduced the concept of stages (which I call build and host) with their respective toolchain (compiler and program, package and pkgconfig databases); and re-used the solver concept of scopes to separate executable dependencies from library dependencies. Executable dependencies (like setup scripts and build-tool dependencies) are solved in the build scope, while library dependencies are solved in the host scope. Roughly speaking, the build scope is for binaries you will run on the build machine while the host scope is for binaries you might want to run elsewhere (e.g. on a different platform).

As you can imagine, this involved a fair amount of work and I am well aware that this PR is too big to be considered as a whole. Furthermore I arbitrarily dropped (obscure) feature and (legacy) workarounds to simplify my work. One example being the "independent goals" solver setting: I do not think anybody is using it (or even should use it) and it was interfering with my rework of the solver scopes. That said, I do not think anything prevents bringing these things back, if only lack of time and effort.

While I am proud of how I was able to make everything fit together nicely, the PR is not without issues (Staged is over-engineered, build stage should be entirely optional, IsGraph is just plain awkward, etc). I welcome (and encourage!) a thorough technical review of the implementation.

This work was prompted by IOG's efforts to build GHC with just cabal-install; and for this reason this branch contains a few commits that are unrelated to cross-compilation. I have spent considerable amount of time making sure each commit corresponds to a single small change[^1]; so it should be easy to identify and separate unrelated, unneeded or unwelcome changes.

Also, shame on me, there are no tests; indeed my only testing was building GHC itself. While some tests for the solver should be easy to add, I do not have a clear testing strategy in mind. Changelog and documentation are not done either.

I have no expectations for this to be merged as-in or even with only minor changes. You are welcome to mark this as draft or to consider it a "implementation included" RFC. I will indeed be offline for the next two weeks, so go wild but I will answer any question or comment only later.

[^1]: and sometimes failed, see the three "all of it" commits.

Here is an itemised list of changes, produced with the help of Claude.

Core Cross-Compilation Features

Stage System:

Add Build and Host stage types to represent compilation targets - 6d741d47823
Implement Staged wrapper type for stage-aware data structures - 6d741d47823
Add stage tracking throughout solver and planning systems - c213cea3417

Toolchain Management:

Introduce Toolchain type with separate compiler, program DB, and package databases - dd1bd2a9e96
Add configToolchain and configToolchains for multi-stage toolchain setup - dd1bd2a9e96
Implement stage-specific program database resolution - ce05b661f1f

Solver Enhancements:

Add stage tracking to SolverId and dependency resolution - c213cea3417
Separate executable dependencies (build scope) from library dependencies (host scope) - 97a950d48f4
Update constraint system to handle stage-specific constraints - 059f001ce1c
Modify package path resolution for cross-compilation scenarios - 97a950d48f4

Major Refactoring

Project Planning (ProjectPlanning.hs):

Complete rewrite of elaboration process for stage-aware planning - 4b55c76abaf
Update instantiateInstallPlan to handle cross-compilation - 69b38ad4999
Refactor dependency resolution to respect stage boundaries - e80cdcb0ae2
Add stage propagation throughout planning pipeline - e80cdcb0ae2

Install Plan:

Enhance InstallPlan to track stage information - f8cec8871a1
Update dependency consistency checking for cross-compilation - 4015ef0d482
Modify plan execution to use appropriate toolchain per stage - 4b55c76abaf

Build System Integration:

Update all build commands (CmdBuild, CmdRepl, CmdExec, etc.) for stage awareness - 4b55c76abaf
Modify executable copying and directory management - 95945f71d17
Add stage-specific build directory handling - 19bebc95da6

Removed/Simplified Features

Legacy Removals:

Remove independent goals solver setting - 5ec63e5ab3e
Drop base-on-base compilation trick - 929676c6127
Remove obsolete build tool workarounds - 086ccc20d85, db76bd1e05f
Eliminate storePackageDBStack mechanism - 929676c6127
Remove in-place building logic - 19bebc95da6

Simplified Logic:

Streamline shared/profiling library selection - f47840db585
Consolidate package database resolution during planning - ce05b661f1f
Remove outdated solver workarounds - db76bd1e05f

Infrastructure Improvements

Error Handling & Logging:

Add HasCallStack constraints throughout codebase - aff15c3a127, 9b7301ca0a6
Enhance logging in build and setup processes - fdf8b596285, e1f578ca533
Improve error messages in solver and validation - a70bb7dd3ca
Add progress tracking with LogProgress - 94e60c4b3cc

Code Quality:

Add Pretty instances for new types - f5a15715212
Improve readability and documentation - 08f604f6843, 4bc268eba20
Add type safety improvements - c303cc064dc
Enhance testing infrastructure - d975d4399d4

New Configuration Options

Build Options:

Add --build-compiler option for specifying build-stage compiler - 085bbd3f566
Extend constraint syntax to support stage qualifiers - 059f001ce1c
Add stage-aware target selection - a8050fcb4b2, 95945f71d17

Project Configuration:

Introduce ProjectConfigToolchain for toolchain settings - dd1bd2a9e96
Add stage-specific package database configuration - ce05b661f1f
Update configuration parsing for cross-compilation options - 97a950d48f4

File Structure Changes

New Modules:

Distribution.Solver.Types.Stage - Core stage definitions - 6d741d47823
Distribution.Solver.Types.Toolchain - Toolchain types - dd1bd2a9e96
Distribution.Client.Toolchain - Toolchain management - dd1bd2a9e96
Distribution.Client.ProjectPlanning.Stage - Stage wrapper types - 4b55c76abaf
Distribution.Client.Types.GenericReadyPackage - Generalized ready packages - e2b7b85a3ea

Major Updates:

27 files in cabal-install-solver for stage-aware solving - 97a950d48f4
40+ files in cabal-install for cross-compilation support - 4b55c76abaf
Multiple test files updated for new functionality - d975d4399d4
CI/workflow updates for cross-compilation testing - Various commits

Cabal Library Enhancements

Feature Additions:

Support for per-file options in extra source files - 20489a78afa
Add generated CMM source support - dfcc65f5b17
Improve response file handling - ab0a509604c
Enhanced program signature tracking - 1023f44e8e8

Build Improvements:

Better GHC integration for cross-compilation - 4f0b69ad5fb
Improved setup script handling - 2439bf7d663
Enhanced package description processing - a6cab47e5b9
Updated build target management - Various commits in the series

Dependency & Build Logic:

Fix setup dependency resolution for correct stage - 332b6eccd09
Improve package self-dependency detection - 86116e103df
Better executable dependency handling - 45172202699
Enhanced build tool dependency management - 65a4fb14651

Check list:

[ ] Patches conform to the coding conventions.
[ ] Any changes that could be relevant to users have been recorded in the changelog.
- [ ] Is the change significant? If so, remember to add significance: significant in the changelog file.
[ ] The documentation has been updated, if necessary.
[ ] Manual QA notes have been included.
[ ] Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

Aug 22 '25 03:08 andreabedini

Looks like very impressive and important work @andreabedini, it's great that you and your team have taken on this work. I'm looking forward to digging into these changes together.

Would it be possible to write a cabal proposal to specify the user-facing changes to this patch? It would be good to get some feedback from other user's interested in cross-compilation and to write down the specification which this patch implements. Once a specification is established it will be easier to assess if the patch implements the agreed specification and there are suitable tests in place.

From your description, it seems that someone could test this patch for themselves by using a wasm or javascript compiler and using `cabal-install" "normally"?

Finally, hope you have a great holiday :)

Aug 22 '25 12:08 mpickering