Patched GHCJS 8.10

[ilyakooo0]

closes #1167

This was inspired by https://github.com/reflex-frp/reflex-platform/tree/develop/haskell-overlays.

Motivation

GHCJS apps are slow and consume a lot of memory. Using patched versions of libraries and runtime yields a significant improvement in both performance and memory usage.

Architectural decisions

From what I have gathered the haskell.nix ecosystem aims to resolve constraints as much as possible from a given hackage snapshot without any easy way of modifying dependencies.

The patched versions of libraries are custom forks and are heavily tied to modification in the boot libraries of GHCJS. These libraries include:

• text
• aeson
• attoparsec

The other libraries can be patched from hackage versions.

The most reasonable way I found of locking down these forked versions of libraries is making them boot libraries of GHCJS. This way they are always used when solving dependencies.

All dependencies of boot libraries need to be themselves boot libraries. This, unfortunately, forced me to add quite a few libraries to GHCJS boot libraries list.

The downsides

There are three real downsides I can think of:

1. The projects will be constrained to using only one specific version of a large number of libraries.
2. Resolving the dependencies of a project with GHC and GHCJS will diverge more than it does now. (It already diverges due to conditionals in a large amount of existing GHCJS-related libraries.)
3. If a project does not use the now boot libraries, then it might yield larger GHCJS executables. (Though this is mostly mitigated by closure-compiler which is widely used)

Other changes

In this PR I also added some optimization flags to GHCJS itself, and the boot libraries. GHCJS itself needs all of the help it can get in the performance department, and the user has no real way to get a more optimized version of boot libraries (though many might want to).

I see no real downside to this change as most users will probably pull these from a cache.

What could be improved

1. The same changes can be applied to GHCJS 8.6, but I did not do it in this PR for time constraint reasons. (Compiling GHCJS takes a long time).
2. The patches should probably be made conditional (allow the user to turn them off), but I couldn't figure out what would be "the correct" way of passing a flag from the user to all of the affected parts.

Testing

I have tested this on two of my projects and I can confirm that everything works.

[hamishmack]

bors try

[iohk-bors[bot]]

try

Timed out.

[ilyakooo0]

On my machine it takes ~2h to boot GHCJS (one derivation).

[angerman]

bors try

[angerman]

On my machine it takes ~2h to boot GHCJS (one derivation).

Yea, don't worry, this will get there eventually :-)

[iohk-bors[bot]]

try

Build failed:

• ci/hydra:Cardano:haskell-nix:required

[ilyakooo0]

Is there something I can do on my end to help merge this?

[angerman]

bors try

[angerman]

@ilyakooo0 we really want to get this merged, holidays have thrown a wrench into getting this merged fast.

[ilyakooo0]

If you can add custom nix caches to your builder, our cache has the GHCJS from my fork: https://app.cachix.org/organization/typeable/cache/typeable

[iohk-bors[bot]]

try

Build failed:

• buildkite/haskell-dot-nix

[hamishmack]

bors try

[iohk-bors[bot]]

try

Timed out.

[angerman]

bors try

[iohk-bors[bot]]

try

Timed out.

[angerman]

bors try

[iohk-bors[bot]]

try

Timed out.

[angerman]

bors try

[iohk-bors[bot]]

try

Timed out.

[ilyakooo0]

In theory we could patch the Hackage package set to hard-code the needed libraries that are not strictly required in the boot library set. This would reduce the time required to boot ghcjs.

[hamishmack]

Would it also reduce the size of the derivation? Timeout is not a problem (bors times out, but hydra keeps going). I think the problem right now is actually Output limit exceeded in https://hydra.iohk.io/build/11433087. IIRC there is a 4GB limit on the sum of all outputs of the derivation on our hydra server.

[angerman]

@hamishmack should we try this for good measure on zw3rk?

[ilyakooo0]

Would it also reduce the size of the derivation?

It should, but I'm not sure by how much.

I am currently time-constrained and will not be able to test this out.

[peterbecich]

edit: question moved to https://github.com/input-output-hk/haskell.nix/issues/1442

---

Question, are all of the non-GHCJS patches https://github.com/input-output-hk/haskell.nix/blob/c3eb75ff6635fc7f8d96ad4e72c79dcd2fdf5fad/overlays/bootstrap.nix#L97 which are here https://github.com/input-output-hk/haskell.nix/blob/c3eb75ff6635fc7f8d96ad4e72c79dcd2fdf5fad/overlays/patches/ghc/ also applied to the GHCJS compiler, in addition to the new patches in this PR? https://github.com/input-output-hk/haskell.nix/blob/c3eb75ff6635fc7f8d96ad4e72c79dcd2fdf5fad/compiler/ghcjs/ghcjs.nix#L20-L25

I'm building a project using ilyakooo0:iko/ghcjs-patch. When compiled with GHCJS, the project produces an error that ought to be fixed by this non-GHCJS patch: https://github.com/input-output-hk/haskell.nix/blob/c3eb75ff6635fc7f8d96ad4e72c79dcd2fdf5fad/overlays/bootstrap.nix#L194

If the non-GHCJS patches are not applied, is it feasible to apply them?

Thanks very much

[ilyakooo0]

@peterbecich I don't think adding new patches overrides old ones, since the value is by default set to an empty list: https://github.com/input-output-hk/haskell.nix/blob/c3eb75ff6635fc7f8d96ad4e72c79dcd2fdf5fad/lib/ghcjs-project.nix#L56

The patches value above is only used here to patch the sources (before the patches you linked above, I think): https://github.com/input-output-hk/haskell.nix/blob/c3eb75ff6635fc7f8d96ad4e72c79dcd2fdf5fad/lib/ghcjs-project.nix#L90-L93

In any case, my patches seem to be applied in a place, distinct from where the bootstrap patches are applied.

---

BTW if you want to use the patched version, you might want to use the iko/ghcjs-patch-master branch. It is updated daily with the commits from upstream. (At the cost of possibly needing to recompile GHCJS)

[peterbecich]

@ilyakooo0 thanks for the detailed response and tip about iko/ghcjs-patch-master. I have reproduced this possible issue using input-output-hk/haskell.nix:master so have determined it is not related to this PR. I've moved my question here: https://github.com/input-output-hk/haskell.nix/issues/1442

[peterbecich]

https://github.com/input-output-hk/haskell.nix/pull/1306#issuecomment-1008276146

I think the problem right now is actually Output limit exceeded in https://hydra.iohk.io/build/11433087.

This thread appears to be discussing the same issue: https://github.com/NixOS/nixpkgs/issues/159612#issuecomment-1043974455

Here is the limit raised from the default 2 GB to 3 GB: https://github.com/NixOS/nixos-org-configurations/pull/206

[ilyakooo0]

@hamishmack @angerman can you trigger CI please?

[hamishmack]

bors try

[iohk-bors[bot]]

try

Timed out.