mirrors.json file is expired

[snoyberg]

• Actions: view https://hackage.haskell.org/mirrors.json
• Expected: signed.expires field will be in the future
• Actual: it's in the past, specifically yesterday

This is resulting in bug reports from users of Stack trying to initialize a new hackage-security-based package index:

• https://github.com/commercialhaskell/stack/issues/3731
• https://github.com/commercialhaskell/stack/issues/4927

I'm currently considering working around this by disabling the expiry checks by default in Stack's usage of hackage-security.

[gbaz]

Should be fixed now?

[gbaz]

confirmed.

[snoyberg]

Yes, the current situation is fixed, but this is the second time we've received bug reports about out of date files. Do you have any method in place to prevent the problem from happening again in the future?

[roberth]

It has expired again. Current value:

{"signatures":[{"keyid":"772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d","method":"ed25519","sig":"/vKO+Kt4025DVUdV0jWUr/EO5MrCrEwacNzH1YdrMj7kdSR0YehjZB96Pe2vr0EI9fflJ6zrOk9Yjl1FdZmqCQ=="}],"signed":{"_type":"Mirrorlist","expires":"2021-01-01T23:59:59Z","mirrors":[{"urlbase":"http://hackage.fpcomplete.com/"},{"urlbase":"http://objects-us-east-1.dream.io/hackage-mirror/"}],"version":7}}

[gbaz]

thanks for the ping, on it. hopefully we can turn key management over to the hackage foundation and professionalize all this in the next year or so. cc @emilypi

[gbaz]

ok, new one should be deployed thanks to duncan.

[roberth]

I can confirm that it works. Thank you gbaz and duncan!

[emilypi]

Cheers! Thanks @roberth and @gbaz

[tp-woven]

I'm seeing this error again: "expires":"2022-06-01T23:59:59Z"

[gbaz]

on it, thanks for the report.

[gbaz]

I am not a holder of the appropriate keys, but I'm reaching out to them now to try to resolve this. Additionally, now that the HF is more stabilized, I will work to try to get this and other keyholding business taken care of by the foundation as a standard place rather than having it distributed in ways that are a bit rather fiddly and can unfortunately fall through the cracks.

[jkoppel]

I am also running into this.

Also, today is the deadline for ICFP artifact evaluation, which makes this particularly dastardly.

[gbaz]

I wish I could help more, but I don't have the keys to perform the resigning. That said, I don't believe cabal install or any recent versions of stack are contingent on this mirrors file, so I am curious as a general proposition what workflows are breaking.

[jkoppel]

artifact@artifact:~/derive-cfg-master$ stack setup
Cabal file info not found for monadIO-0.11.1.0, updating
Selected mirror https://hackage.haskell.org/
Downloading timestamp
Downloading snapshot
Downloading mirrors
Verification error: <repo>/mirrors.json is expired
Downloading root
Waiting to acquire cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Downloading timestamp
Downloading snapshot
Downloading mirrors
Verification error: <repo>/mirrors.json is expired
Downloading root
Waiting to acquire cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Downloading timestamp
Downloading snapshot
Downloading mirrors
Verification error: <repo>/mirrors.json is expired
Downloading root
Waiting to acquire cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Downloading timestamp
Downloading snapshot
Downloading mirrors
Verification error: <repo>/mirrors.json is expired
Downloading root
Waiting to acquire cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Downloading timestamp
Downloading snapshot
Downloading mirrors
Verification error: <repo>/mirrors.json is expired
Downloading root
Waiting to acquire cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Acquired cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Released cache lock on /home/artifact/.stack/pantry/hackage/hackage-security-lock
Verification loop. Errors in order:
  <repo>/mirrors.json is expired
  <repo>/mirrors.json is expired
  <repo>/mirrors.json is expired
  <repo>/mirrors.json is expired
  <repo>/mirrors.json is expired

[dmvianna]

@jkoppel there is a workaround. https://github.com/commercialhaskell/stack/issues/3731#issuecomment-507125622

[istathar]

We're encountering this as well:

[CGenie]

Let's see. This issue is commented around the following days :

• 2019/06/01
• 2021/01/01
• 2022/06/01

Seems that at least every 1.5 years something gets broken in the hackage server.

[dcoutts]

I'll fix this now by re-signing the mirrors file. Will provide an update here when it's done.

But yes it would be wise to hand out a couple more operational keys. That'll need the root key holders to re-sign.

[dcoutts]

Ok, next step is to upload it:

{
  "signatures": [{"keyid":"280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833","method":"ed25519","sig":"9YaOmIr/m1oSrYkITopMthF8pExyqmjHCDXPytt2DqDYB/5NlYUDXtXNr1Y1yTKglLNEE2+c720N2un0lnEQCg=="}],
  "signed": {
    "_type": "Mirrorlist",
    "expires": "2023-06-01T23:59:59Z",
    "mirrors": [
      {
        "urlbase": "http://hackage.fpcomplete.com/"
      },
      {
        "urlbase": "http://objects-us-east-1.dream.io/hackage-mirror/"
      }
    ],
    "version": 9
  }
}

(It can be canonicalised or not, doesn't matter except file size)

[dcoutts]

BTW, I agree giving operational keys to folks from the HF is a good idea. The root key holders will need to issue and sign an updated root.json that lists the operational keys (mirrors, snapshot, timestamp etc).

It might also be a good opportunity to add more root key holders & drop some who are no longer active.

[ketzacoatl]

@dcoutts / @gbaz / etc

While having more key holders would help in a pinch, the bigger problem is that no one is tracking this task on a calendar to do the resign before the expiration. Just like with SSL/TLS certs, someone needs this task on a calendar with reminders, and we need to do the resign before the expiration. Adding key holders is nice, but we should first figure out the process for how that task is tracked and who is responsible for the tracking. We should not be waiting until the expiration has passed.

EDIT: It's really frustrating that I'm repeating what was stated back in 2019 (almost 3 years ago!!):

Yes, the current situation is fixed, but this is the second time we've received bug reports about out of date files. Do you have any method in place to prevent the problem from happening again in the future?

[pxeger]

Perhaps the expiry times could be made shorter: that way, people will be more used to doing it regularly, and might even think to partially automate it.

[gbaz]

Ok, this is now resolved again for the time being. I agree the process is very bad, and that changes were never made to the procedures to keep this from reoccuring. I'm going to follow up immediately with the HF to try to put this into a better situation -- especially since we need a root resigning in approx the next 60 days as well.

[dcoutts]

@ketzacoatl you're not wrong! :smiley:

[tp-woven]

Thanks for fixing this, I can confirm that it works for me now without the workaround. :)

[avdv]

I'm currently considering working around this by disabling the expiry checks by default in Stack's usage of hackage-security.

Since this issue is still open and it was not obvious to me, please note that the checks indeed have been disabled in newer Stack versions, by commit commercialhaskell/stack@4d54687 which was released in Stack 2.1.3.

If you come here because you're using Stack, you might see this error because you're using an old version.

[jkoppel]

The machine I was working on has been taken down, but I both had a fresh install of Stack and had run stack upgrade.

(Although I don't remember if stack upgrade just outright failed because of this issue.)

[Kleidukos]

@gbaz shall we close this issue?

[istathar]

Have the key holders been updated and the process to ensure keys are rotated been transferred to Haskell Foundation yet?

[Kleidukos]

@istathar The Haskell Foundation executive is now responsible for coordinating the process, although the process itself has not be reformed yet.