CI job "Build & Test - Cross" fails

[avdv]

Describe the bug Most recently, the "Build & Test - Cross" job has started to fail, e.g. here.

To Reproduce Run the Continuous Integration workflow.

Expected behavior It should finish successfully.

Environment Github Actions ubuntu-latest, commit c5335e62829e0e08d7986457a97eb94886081d78

Additional context It seems the aarch64 GHC is no longer found in a cache and thus it starts to build it, which requires GHC 8.8.4 to be build first:

building '/nix/store/3jch2jzcyhp5v110r8bvbcdka7lgsy98-source.drv'...

trying https://github.com/input-output-hk/hackage.nix/archive/8299f5acc68f0e91563e7688f24cbc70391600bf.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 48.8M    0 48.8M    0     0  3843k      0 --:--:--  0:00:13 --:--:-- 4755k
unpacking source archive /build/8299f5acc68f0e91563e7688f24cbc70391600bf.tar.gz
these 35 derivations will be built:
  /nix/store/2skjk7lw8f32gv96b4anaxp10igfnd4g-default-Setup-src.drv
  /nix/store/wg8j99nxmzxi56dcjfdcingzl1fg6v8c-ghc-8.4.4.drv
  /nix/store/7sfka6vsvrjywfqpm7580nalqhsd0d6a-default-Setup-setup-config.drv
  /nix/store/dcii9aalm2jwaqrmpjasdsssgx73xqxh-default-Setup-setup.drv
  /nix/store/dwk6y9cjyam2nnk4qfgz4vgr7rgfx92x-hscolour-exe-HsColour-1.24.4-config.drv
  /nix/store/hsmwgxb1vq44i1v9rz6r7hnblg5nrh24-hscolour-exe-HsColour-1.24.4-ghc-8.4.4-env.drv
  /nix/store/iz0gy65xb1v4n9w39cyv19kgbh0z1k5l-hscolour-1.24.4-src.drv
  /nix/store/fvfcgrpz6ywazyrh965q57zash777knd-hscolour-exe-HsColour-1.24.4.drv
  /nix/store/6b3zmfsnf8k9zc4alg131s5g1cwfy91g-alex-exe-alex-3.2.4-config.drv
  /nix/store/47sk42cy5392jjffn6ybz41qgmg6jbsa-alex-exe-alex-3.2.4-ghc-8.4.4-env.drv
  /nix/store/ql1k1s8wsq6w30jcaadsxmcdqjxrj640-alex-3.2.4-r1-src.drv
  /nix/store/gkwx18cy5vdqizar77qbsnzxrlasqzjg-alex-exe-alex-3.2.4.drv
  /nix/store/rq2z93qz3vwv7sd5r3r1zampcnci8qvr-happy-exe-happy-1.19.12-config.drv
  /nix/store/8vzkdkh0i0hxzsq5x4m633riv0iaa6q4-happy-exe-happy-1.19.12-ghc-8.4.4-env.drv
  /nix/store/hdk8v5ffv88pq999rl60yhsw4vb24f17-happy-1.19.12-r1-src.drv
  /nix/store/nnsrd4jsa3q1z9j61qzk0f04xpc9yxn8-happy-exe-happy-1.19.12.drv
  /nix/store/82gngnf273fkhcjnlvyh8f23rzicg1i0-ghc-8.8.4.drv
  /nix/store/y1jjrp8h0aqn7w82f82cdqj3q61hpmbk-ghc-8.8.4-x86_64-fedora27-linux.tar.xz.drv
  /nix/store/3msl59bbcmygjm9l5q079mi9b38d1cyq-ghc-8.8.4.drv
  /nix/store/yqzkrzs4a1h9qwbs5rflr05cciz5jic8-happy-exe-happy-1.19.12-config.drv
  /nix/store/ch0m2f2wii47ayyky80jr30nsc1illsb-happy-exe-happy-1.19.12-ghc-8.8.4-env.drv
  /nix/store/80s56aa61kbsllwii73wqps5vim0wfby-default-Setup-setup-config.drv
  /nix/store/jy193p4l9frrrg1nknvgy02cfag4xsc0-default-Setup-setup.drv
  /nix/store/8bs0pb0x757kk9ms9p9pn8l1dim0198h-happy-exe-happy-1.19.12.drv
  /nix/store/3q7nfpkjg8i107h2z9fgm5sybksy24jh-hscolour-exe-HsColour-1.24.4-config.drv
  /nix/store/hx3fvrilr789np9iagx5j0b5yglcfhsx-hscolour-exe-HsColour-1.24.4-ghc-8.8.4-env.drv
  /nix/store/gdc3k36gcxnf7spb3cvvx4l1fkslmrhb-hscolour-exe-HsColour-1.24.4.drv
  /nix/store/abii8a85fwmdmlviganzykmp85q3fia5-alex-exe-alex-3.2.4-config.drv
  /nix/store/2kqdlanda3fflkb3wn28463v1hgjnf0z-alex-exe-alex-3.2.4-ghc-8.8.4-env.drv
  /nix/store/h2v1r1ri4sf0zrxdj2cdjiwdngvr98jq-alex-exe-alex-3.2.4.drv
  /nix/store/1rj7s6j05nc5cx6fh81q1rpyfinkrwv8-aarch64-unknown-linux-gnu-ghc-8.10.7.drv
  /nix/store/j2jlyvgavg8783aclf72frz7zai520rj-iserv-wrapper.drv
  /nix/store/gxh1jiqxqsgivz1x1x486wsinli18sd5-ghc-llvm-wrapper.drv
  /nix/store/wx605fq7b390kfwy4d3xyj1724kz21r2-ghc-aarch64-symlinks.drv
  /nix/store/5x8rqr3gxg9h40s0a0aixczwddpv7hyd-ghc-aarch64-env.drv

and then it fails after ~1h30m with

copyFile: resource exhausted (No space left on device)

The aarch64 GHC package used to be available from iog:

copying path '/nix/store/ga2whkmxan67sq6l64zxapwzj12gkfk2-aarch64-unknown-linux-gnu-ghc-8.10.7' from 'https://cache.iog.io/'...

It vanished from the cache:

$ curl --fail https://cache.iog.io/ga2whkmxan67sq6l64zxapwzj12gkfk2.narinfo
curl: (22) The requested URL returned error: 404

[avdv]

To fix this issue, we would either have to upgrade GHC (which is blocked on https://gitlab.haskell.org/ghc/ghc/-/issues/21767 -- would need to check if this is still the case with a more recent GHC), setup a binary cache we can use were we can get a pre-build GHC amd64 -> aarch64 cross compiler.