haskell
hackage-mirror stops on server-500 error
When running hackage-mirror, the client eventually hits hermes-1.3.4.3, which doesn't seem to have a file on hackage. The mirror client then stops mirroring for half an hour (due to --continuous) and then tries again, failing in the same way:
$ hackage-mirror servers.cfg --continuous -vvv
Selected mirror http://hackage.haskell.org
Downloading timestamp
Sending:
GET /timestamp.json HTTP/1.1
Host: hackage.haskell.org
User-Agent: hackage-mirror/0.5.1
Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Cache-Control: public, no-transform, max-age=60
Content-MD5: U+wGwY5oXtD8pDDn/hX6dw==
ETag: "53ec06c18e685ed0fca430e7fe15fa77"
Content-Length: 464
Accept-Ranges: bytes
Date: Tue, 22 Feb 2022 18:36:16 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-lax10658-LGB
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1645554976.957252,VS0,VE97
Vary: Accept-Encoding
Waiting to acquire cache lock on /runtime/mirror-cache/hackage/hackage-security-lock
Acquired cache lock on /runtime/mirror-cache/hackage/hackage-security-lock
Released cache lock on /runtime/mirror-cache/hackage/hackage-security-lock
No cached index available for target repo
downloading http://admin:...@localhost:8080/packages/index.tar.gz to
mirror-cache/mirror/00-index.tar.gz
Sending:
GET /packages/index.tar.gz HTTP/1.1
Host: localhost:8080
User-Agent: hackage-mirror/0.5.1
If-None-Match: "7896f4cd4a536563d5509acff1bdf0ff"
Authorization: Basic YWRtaW46YWRtaW4=
Creating new connection to admin:admin@localhost:8080
Received:
HTTP/1.1 304 Not Modified
Cache-Control: public, no-transform, max-age=300
Connection: Keep-Alive
Date: Tue, 22 Feb 2022 18:36:22 GMT
ETag: "7896f4cd4a536563d5509acff1bdf0ff"
Server: Happstack/7.7.1.2
mirror-cache/mirror/00-index.tar.gz unchanged with ETag
7896f4cd4a536563d5509acff1bdf0ff
87303 packages to mirror. Ignoring 15849 package(s) that cannot be mirrored
(for details see mirror-cache/missing-hackage and
mirror-cache/unmirrorable-hackage-mirror)
mirroring hermes-1.3.4.3
Selected mirror http://hackage.haskell.org
Downloading package hermes-1.3.4.3
Sending:
GET /package/hermes-1.3.4.3.tar.gz HTTP/1.1
Host: hackage.haskell.org
User-Agent: hackage-mirror/0.5.1
Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/plain; charset=UTF-8
Location: /package/hermes-1.3.4.3/hermes-1.3.4.3.tar.gz
Content-Length: 0
Accept-Ranges: bytes
Date: Tue, 22 Feb 2022 18:36:26 GMT
Via: 1.1 varnish
Age: 2382
Connection: keep-alive
X-Served-By: cache-lax10656-LGB
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1645554987.548999,VS0,VE1
301 - redirect
Redirecting to
http://hackage.haskell.org/package/hermes-1.3.4.3/hermes-1.3.4.3.tar.gz ...
Sending:
GET /package/hermes-1.3.4.3/hermes-1.3.4.3.tar.gz HTTP/1.1
Host: hackage.haskell.org
User-Agent: hackage-mirror/0.5.1
Creating new connection to hackage.haskell.org
Received:
HTTP/1.1 500 Internal Server Error
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Fastly-Restarts: 1
Transfer-Encoding: chunked
Accept-Ranges: bytes
Date: Tue, 22 Feb 2022 18:36:26 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-lax10679-LGB
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1645554987.648724,VS0,VE166
Content-Length: 445
Warning: Failed to download package hermes-1.3.4.3,
Unexpected response (5,0,0)for http://hackage.haskell.orgpackage/hermes-1.3.4.3.tar.gz
Abandoning this mirroring attempt.
Next try will be in 33 minutes, at 19:09 UTC
Current output from http://hackage.haskell.org/package/hermes-1.3.4.3.tar.gz (note that the URL listed in the "unexpected response" above is missing a /):
Happstack 7.7.1
Something went wrong here
Internal server error
Everything has stopped
The error was "/home/hackage/hackage.prod/state/blobs/a4/a40900b9b0da3715595660c9ede6df77: openBinaryFile: does not exist (No such file or directory)"
That package was removed from hackage because it was a proprietary package mistakenly uploaded. I'd welcome a PR to make the mirror client robust to such missing things.
I'll look into making the client robust to this sort of failure. However, I'm hesitant to make the client ignore server 500 errors. If the issue is that the server is actually down due to some internal failure, continuing to request packages could add unnecessary load to the server. Granted, there may not be very many mirror clients running out there?
Should the server also 404 instead of 500 in this case? Or should hermes itself be removed from the package index? Or perhaps since https://hackage.haskell.org/package/hermes returns a 451 (Unavailable For Legal Reasons) code, maybe the individual versions should also return 451?