Improve maintainability of hydra build toolchain

[locallycompact]

Why

The Hydra (protocol) developers need a convenient way to build and update dependencies of the project. This also includes provisioning of development tools in the right version (aka dev shell), static builds, and docker images.

Furthermore developers wanting to use packages, should be able to easily depend our libraries and get started quickly.

While our primary way to use the project should remain using nix, we want to keep support for a non-nix way to build and develop using cabal only (with some caveats on getting the right versions of tools).

What

• Dynamic and static binaries (for linux), as well as docker images can (still) be built

• Not need to create a build plan from scratch when initially building the project.

  • nix build does not use a solver to find a build plan
  • cabal build effectively just uses a pre-selected set of packages as well (could be trivial constraints - already solved)

• Updating dependencies workflow is changed

• Downstream projects can depend on our libraries

  • use a flake input to get our packages (nix users)
  • copy our cabal.project constraints (cabal users)

• Should be able to update the build plan for all users

  • by updating the nix package set (nix users)
  • TBD: How to do this for cabal-only users (who do not have nix installed)

• The nix flake outputs are fully inspectable for all / on all platforms. i.e. nix flake show and nix flake check just work

How

• Avoiding IFD by using horizon instead of haskell.nix (which calls cabal in an IFD)

• TBD: Persist horizon package set into cabal.project as a set of constraints (or use cabal freeze / lock files?)

• TBD: cross-toolchain-compilation with muslc still needs to be possible

• TBD: hoogle with all dependencies included in the index in the dev shell

• TBD: a working HLS (currently patched, any latest is fine)

• TBD: check whether all haskell.nix modules are still doable with horizon

---

Detailed rationale

Our nix setup is causing a lot of complications both for us and for downstream consumers of hydra libraries. The most serious of these is:

The presence of IFD when constructing the dependency graph means the flake can not be evaluated statically. Consequently it is very hard to debug attribute values when an evaluation fails, particularly when debugging the expression for a non-native system.

There are also several quality of life issues that I believe should also be addressed.

1. Implicitly chosen dependencies mean that downstream consumers of hydra have to solve build plans themselves in addition to actually developing their application. Since we have solved this build plan ourselves, we should be able to share this result with them as an input value to their project directly, but we can't. Since IFD will recompute the build plan each time, any flake update the user makes can cause build issues that they themselves can't debug.

2. We also have the same problem when trying to resolve compatible versions of our own dependencies, that others upstream have already partially solved. We essentially redo this work every time.

3. We can not derive our CI directly from the flake, as we can not evaluate it statically. With a static flake, we could turn the flake into CI jobs purely, as well as including linting checks like fourmolu.

4. We can not detect breakages to reverse dependencies automatically. For example, if a bump to cardano-api were to break hydra.

I propose Horizon Haskell Stable Package Sets as a solution to these issues. Horizon Haskell uses explicitly pre-computed stable package sets as the input to a build plan that can be evaluated statically without IFD. We can use this package set as an input both to our own nix code, and offer it to consumers of our packages containing a collection of libraries that all work together. We would also have a package set repository (horizon-hydra) that could receive pull requests, creating a common place for all consumers to contribute to advancing the build plan forward, sharing the work of finding working build plans and reifying them explicitly as a common input.

Horizon stable package sets are also maintainable by non-nix users with testimonials to support that claim. (They will be on the website soon.)

Drawbacks: Some of the features of haskell.nix are not yet supported by horizon, such as static builds and windows cross compilation. Static builds are likely not going to be an issue, and I would potentially suggest not solving windows XC with nix at all. We could compile horizon's data model directly to a cabal.project and compile natively.

There is an example branch here, with instructions. It is incomplete, but the remaining work is just more of the same - as well as initializing a new horizon-hydra repository as an extension to horizon-plutus.

https://github.com/input-output-hk/hydra/commits/lc/horizon-example