Invalid hackage-repo-hackage.haskell.org-at in cache

[hamishmack]

When the hackage-repo-hackage.haskell.org-at-X derivation is built it:

Links the file into the output derivation https://github.com/input-output-hk/haskell.nix/blob/3beaf26583d268e08707e0ad2ba9b186b33dc61a/mk-local-hackage-repo/default.nix#L24

It hashes the same file https://github.com/input-output-hk/haskell.nix/blob/3beaf26583d268e08707e0ad2ba9b186b33dc61a/mk-local-hackage-repo/default.nix#L25-L27

Then sets the md5 and sha256 sums in snapshot.json based on the hash of the index file https://github.com/input-output-hk/haskell.nix/blob/3beaf26583d268e08707e0ad2ba9b186b33dc61a/mk-local-hackage-repo/default.nix#L39.

So the hash of the linked 01-index.tar.gz file should always match what is in the snapshot.json file.

However this is not the case for the cached copy of this derivations output for 2020-10-16T000000Z on hydra.

$ cat /nix/store/k6w9kkvgm0835in4v7d1qdkifd62vw5x-hackage-repo-hackage.haskell.org-at-2020-10-16T000000Z/snapshot.json 
{
  "signatures": [],
  "signed": {
    "_type": "Snapshot",
    "expires": "4000-01-01T00:00:00Z",
    "meta": {
      "<repo>/01-index.tar.gz": {
        "hashes": {
          "md5": "1f99b1d9ccfe67c1defc916ec8c38695",
          "sha256": "8dbf38995877392a0391fe8bb4cdd272a785924e23a3512bfd01cac563ce2040"
        },
        "length": 95185648
      },
      "<repo>/root.json": {
        "hashes": {
          "md5": "4b0d662d79d58e2df360ac62a6c2f103",
          "sha256": "90e85d626c9d75c13a5874fe364c2a42920f69c70df7917c37b52c1b8988add1"
        },
        "length": 547
      },
      "<repo>/mirrors.json": {
        "hashes": {
          "md5": "d70732b3e90e821e5d36837dc35bcd5e",
          "sha256": "ad173f7f47bcaafb1ae79b1b72677aeb518e8e926a43f0ac5a4dbcbd00d01eeb"
        },
        "length": 145
      }
    },
    "version": 1
  }
}

$ nix-hash --flat --type md5 /nix/store/k6w9kkvgm0835in4v7d1qdkifd62vw5x-hackage-repo-hackage.haskell.org-at-2020-10-16T000000Z/01-index.tar.gz 
691c269ca9e09d1b0d3ea47b53d15c86

This leads to the following error when cabal new-update is run in the dot-cabal-hackage.haskell.org-at derivation:

Verification loop. Errors in order:
  Invalid hash for <repo>/01-index.tar.gz
  Invalid hash for <repo>/01-index.tar.gz
  Invalid hash for <repo>/01-index.tar.gz
  Invalid hash for <repo>/01-index.tar.gz
  Invalid hash for <repo>/01-index.tar.gz

builder for '/nix/store/v5x0j5c8p4w3qlb9q747xkznb50kl1yp-dot-cabal-hackage.haskell.org-at-2020-10-16T000000Z.drv' failed with exit code 1

The index is a fixed output derivation https://github.com/input-output-hk/haskell.nix/blob/3beaf26583d268e08707e0ad2ba9b186b33dc61a/overlays/haskell.nix#L187-L195

And the contents of the linked file do seem to be correct (even when downloaded from hydra). Somehow the hash values snapshot.json came out wrong (the length of the file is correct).

The question is how does this happen? Perhaps the file was being copied to the store of the machine running the derivation and had not been written?

Steps to reproduce on linux run this with hydra binary cache is in the substituters:

nix-store --delete /nix/store/k6w9kkvgm0835in4v7d1qdkifd62vw5x-hackage-repo-hackage.haskell.org-at-2020-10-16T000000Z
nix-store --delete /nix/store/ry4ija9ql90pjp1i703f488wflsqcm0f-hello-plan-to-nix-pkgs
nix-build -E '((import ./. {}).pkgs.haskell-nix.tool "ghc8102" "hello" { version = "1.0.0.2"; index-state = "2020-10-16T00:00:00Z"; }).project.plan-nix' --no-out-link

Run it --substituters '' to see what should have happened.

[miuirussia]

After this fix I can't build my project: https://github.com/miuirussia/nix-dotenv/runs/1353937376?check_suite_focus=true

/nix/store/9lvbs7p2v4z5l2djnq1gri8fskbz6g41-hackage-repo-hackage.haskell.org-at-2020-10-21T011410Z/root.json: openBinaryFile: does not exist (No such file or directory)

[hamishmack]

Similar error started happening yesterday on hydra for our nightly hackage and stackage updates.

trace: Shell for pkg
trace: Shell for pkg
trace: Using index-state: 2020-05-31T00:00:00Z for hoogle
trace: To make this a fixed-output derivation but not materialized, set `plan-sha256` to the output of /nix/store/8968lk0c3hadbsbh5sxx816a7b474yf1-calculateSha
trace: To materialize the output entirely, pass a writable path as the `materialized` argument and pass that path to /nix/store/qpwaj1w3h42l8nd89bsq10jwf4byp5id-generateMaterialized
trace: Using index-state: 2020-05-31T00:00:00Z for hoogle
trace: To make this a fixed-output derivation but not materialized, set `plan-sha256` to the output of /nix/store/7nl26cgnmcnhxv3s7jib17qmfjhp5bkf-calculateSha
trace: To materialize the output entirely, pass a writable path as the `materialized` argument and pass that path to /nix/store/49i155m4l4n7a7a32blr81cvlqlmirjd-generateMaterialized
trace: Using latest index state for cabal-install!
trace: Using index-state: 2020-11-05T00:00:00Z for cabal-install
building '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' on 'ssh://[email protected]'...
/nix/store/4cc8xywcb2gi6fld5la679h8rh1dqwz1-hackage-repo-hackage.haskell.org-at-2020-11-05T000000Z/root.json: openBinaryFile: does not exist (No such file or directory)
builder for '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' failed with exit code 1
error: build of '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' on 'ssh://[email protected]' failed: builder for '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' failed with exit code 1
builder for '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' failed with exit code 1
cannot build derivation '/nix/store/m3zl9wvazld1dc7w2cpkc11dcrjcz6gr-cabal-install-plan-to-nix-pkgs.drv': 1 dependencies couldn't be built
building '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' on 'ssh://[email protected]'...
/nix/store/4cc8xywcb2gi6fld5la679h8rh1dqwz1-hackage-repo-hackage.haskell.org-at-2020-11-05T000000Z/root.json: openBinaryFile: does not exist (No such file or directory)
builder for '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' failed with exit code 1
error: build of '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' on 'ssh://[email protected]' failed: builder for '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' failed with exit code 1
builder for '/nix/store/wdw7ygmcxn1rw41nsabz685wlwbc4f1v-dot-cabal-hackage.haskell.org-at-2020-11-05T000000Z.drv' failed with exit code 1
cannot build derivation '/nix/store/m3zl9wvazld1dc7w2cpkc11dcrjcz6gr-cabal-install-plan-to-nix-pkgs.drv': 1 dependencies couldn't be built
error: "build of '/nix/store/m3zl9wvazld1dc7w2cpkc11dcrjcz6gr-cabal-install-plan-to-nix-pkgs.drv' failed"
error: unexpected EOF reading a line

[shajra]

I'm not sure if anyone else faced this, but I was having a problem when pinning the index-state of Hackage. I'd do my normal trick of putting in a wrong hash and then seeing what hash Nix told me to use instead.

But then this lead me to the check that @hamishmack put in that pointed to this issue, and I was stuck for a while.

For me, the work-around was to not use the hash that Nix reported back, but rather to use the hash that comes from this file, which is what we check against:

https://github.com/input-output-hk/hackage.nix/blob/master/index-state-hashes.nix

Eventually, when this bug is resolved, these two hashes should be the same. I just didn't know til now where to get the right hash until I found that file.

[andreabedini]

@hamishmack is this still relevant? I don't think errors like

/nix/store/9lvbs7p2v4z5l2djnq1gri8fskbz6g41-hackage-repo-hackage.haskell.org-at-2020-10-21T011410Z/root.json: openBinaryFile: does not exist (No such file or directory)

have anything to do with hashes. Do we have a repro?