aarch64-darwin: Passing `system = "x86_64-darwin"` still triggers build of aarch64 GHC

[JackKelly-Bellroy]

We tried to use haskell.nix on an M1 Mac with Rosetta 2 installed, by overriding the system parameter:

let
  # NB: We are _not_ using flakes here
  pkgs = (import sources."haskell.nix" { system = "x86_64-darwin"; }).pkgs;
  project = pkgs.haskell-nix.cabalProject { ... };
in
  project.shellFor { ... }

This still triggers a build of GHC for aarch64-darwin, which even given M1 performance takes a very long time. I suspect that this is due to the eval-packages.nix overlay forcing the use of builtins.currentSystem, when building various tools for the IFDs:

https://github.com/input-output-hk/haskell.nix/blob/f44f0845b29bf9caf7884510f4b17e9423673fc9/overlays/eval-packages.nix#L9-L13

I'm not clear on the best way to fix this - we certainly don't want to plumb system through to this import, as that would break cross-compilation. We could:

1. Add another parameter which can override the system detection in eval-packages.nix?
2. Special-case system = "x86_64-darwin" when builtins.currentSystem == "aarch64-darwin"?
3. Something else?

I'm happy to try and PR this, but I'll need some guidance.

CC @Tristano8

[angerman]

Ohh, that's a nasty bug indeed. We've probably not hit this before because there were only very few systems that would support multiple archs at the same time.

[angerman]

What happens if we drop the builtins.currentSystem outright and go always with final.buildPackages.system?

[michaelpj]

The whole reason for the eval-packages dance is so that when you are cross-compiling from X->Y, you can still evaluate on X! Otherwise you simply can't e.g. evaluate a darwin cross derivation on a linux machine, which really sucks. The typical usecase is "I don't have a darwin builder (hydra will do that when I push) but I want to check that everything evaluates properly locally". (In fact, I think this was added because I complained about it :) )

In many ways it would be simpler and more logical to do the IFDs on the build platform, but it does make developing miserable.

[JackKelly-Bellroy]

We've found a workaround for ARM Macs: set system = x86_64-darwin in nix.conf, rather than passing it as an argument (and optionally, extra-platforms = aarch64-darwin). This has the sad side-effect of forcing Rosetta 2 for everything else you get from Nix, but gets you IFD evals without building GHC, and without forcing rebuilds of everything else.

I agree that eval-packages exists for a good reason, and for >=99% of non-flake uses, builtins.currentSystem is the right thing to use. I think this problem goes away when one of two things happen:

1. IOHK Hydra gets an aarch64-darwin builder (so you can evaluate IFD without building AArch64 GHC, but probably removes a lot of the reason to pass system = "x86_64-darwin" to haskell.nix?)
2. We add a parameter somewhere which can override the system used by evalPackages.

[angerman]

@JackKelly-Bellroy you can try

https://cache-1.zw3rk.com/
loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=

as a binary cache. (At some point I should generate a new key called zw3rk or something.

https://ci.zw3rk.com does have 3 m1 minis and does build haskell.nix as well (as well as a ton of other stuff).

[lorenzo]

@angerman I tried this cache myself and it still tries to build GHC on my M1 Mac

[G3zz]

Hi, are m1 macs still unsupported by the binary cache?

I've added the below to my darwin-configuration.nix file:

# Binary Cache for Haskell.nix  
  nix.binaryCachePublicKeys = [
    "hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ="
    "loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk="
  ];
  nix.binaryCaches = [
    "https://cache.iog.io"
    "https://cache.zw3rk.com"
  ];   

I am trying to use ghc901 in a simple hello-world project (flake below):

{
  inputs.haskellNix.url = "github:input-output-hk/haskell.nix";
  inputs.nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  outputs = { self, nixpkgs, flake-utils, haskellNix }:
    flake-utils.lib.eachSystem [ "aarch64-darwin" ] (system:
    let
      overlays = [ haskellNix.overlay
        (final: prev: {
          # This overlay adds our project to pkgs
          helloProject =
            final.haskell-nix.project' {
              src = ./.;
              compiler-nix-name = "ghc901";
              # This is used by `nix develop .` to open a shell for use with
              # `cabal`, `hlint` and `haskell-language-server`
              shell.tools = {
                cabal = {};
                hlint = {};
                haskell-language-server = {};
              };
              # Non-Haskell shell tools go here
              shell.buildInputs = with pkgs; [
                nixpkgs-fmt
              ];
              # This adds `js-unknown-ghcjs-cabal` to the shell.
              # shell.crossPlatforms = p: [p.ghcjs];
	      shell.withHoogle = true;
            };
        })
      ];
      pkgs = import nixpkgs { inherit system overlays; inherit (haskellNix) config; };
      flake = pkgs.helloProject.flake {
        # This adds support for `nix build .#js-unknown-ghcjs-cabal:hello:exe:hello`
        # crossPlatforms = p: [p.ghcjs];
      };
    in flake // {
      # Built by `nix build .`
      defaultPackage = flake.packages."hello:exe:hello";
    });
}

I can see that it is querying the cache at https://cache.zw3rk.com, yet it still seems to try to build GHC:

Frustratingly, some time later I get a failure:

[0/1 built, 0.0 MiB DL] querying easy-file-lib-easy-file-0.2.2-haddock-doc on https://cache.zw3rk.com
error: builder for '/nix/store/78p7gbqi2lswl144r5zjmjzjgplnkxk4-ghc-lib-parser-lib-ghc-lib-parser-9.2.3.20220527.drv' failed with exit code 1;
       last 10 log lines:
       > 'vortex' is not a recognized processor for this target (ignoring processor)

[denizdogan]

I am asking myself the same question as @G3zz before me, is aarch64-darwin expected to be found in the binary caches at this time? I can't seem to get it to work no matter what I do.

[G3zz]

I tried it not using flakes and I got a cache hit - something in the flake causes a cache miss

[denizdogan]

@G3zz Are you 100% sure you’re using the same configuration and version of Haskell.nix? Could you please provide some example code so that I can see if I can reproduce it?

[hamishmack]

I don't think we build ghc901 for all systems (even on ci.zw3rk.com). Have you tried switching to ghc902?

nix-diff is really good in these situations. If you can get the .drv for pkgs.compiler.ghc902 on the cached and non cached versions then nix-diff should hopefully reveal what is different.

[hamishmack]

I'm not clear on the best way to fix this - we certainly don't want to plumb system through to this import, as that would break cross-compilation. We could:

1. Add another parameter which can override the system detection in eval-packages.nix?
2. Special-case system = "x86_64-darwin" when builtins.currentSystem == "aarch64-darwin"?
3. Something else?

We removed overlays/eval-packages.nix and instead evalPackages is now an optional argument you can pass to the project. You can also pass just evalSystem = "x86_64-darwin" to the project (and the default evalPackages will then use that).

Because builtins.currentSystem requires --impure the default evalSystem is pkgs.buildPackages.system.

Flakes must make a choice of evalSystem (or system for evalPackages) and the options are:

• Pass nothing and it will match buildPackages.system (if the flake has multiple systems you will need builders for them all to run nix flake show).
• Hard code it to some system you know you will have a builder for.
• Pass in builtins.currentSystem.
• Some logic like evalSystem = if system == "aarch64-darwin" then "x86_64-darwin" else system; or evalSystem = if builtins.currentSystem == "aarch64-darwin" then "x86_64-darwin" else system;.

If you choose to use builtins.currentSystem keep in mind:

• You will have to pass --impure to nix.
• This will prevent the nix from caching the results of evaluations.
• You might want to check how long things like nix develop take to launch.

If you do use builtins.currentSystem then you can pass --system x86_64-darwin on the command line or add system = x86_64-darwin to ~/.config/nix/nix.conf and that will cause make it so everything runs as if you had a x86 only system with an aarch64 builder.

% nix repl --system x86_64-darwin
Welcome to Nix 2.12.0pre20220921_f704c27. Type :? for help.

nix-repl> builtins.currentSystem
"x86_64-darwin"

nix-repl> :q
% nix repl --system aarch64-darwin
Welcome to Nix 2.12.0pre20220921_f704c27. Type :? for help.

nix-repl> builtins.currentSystem
"aarch64-darwin"

nix-repl> :q