firefox-nightly broken: checksums missing for x86-64

[colemickens]

I've been getting this when updating for the last few days:

error: unable to download 'https://archive.mozilla.org/pub/firefox/nightly/2019/07/2019-07-06-09-34-01-mozilla-central/firefox-69.0a1.en-US.linux-x86_64.checksums': HTTP error 404

https://archive.mozilla.org/pub/firefox/nightly/2019/07/2019-07-06-09-34-01-mozilla-central/

[colemickens]

Then again, when trying to pin an older version, I failed and just re-updated to latest and it worked this time. Maybe I'm getting unlucky and updating before the final checksums are uploaded?

[rail]

Yeah, this is a race condition. The json file (buildhub.json) we use to find the latest build is published before the checksums are published. I hit this a few times in the past. Not sure if this is something very important though. :)

cc @MihaiTabara

[MihaiTabara]

Yeah. buildhub.json is transferred along with the en-US builds, which is among the first beetmover jobs to run. Then there's checksums-signing and then beetmover-checksums. There's at least few minutes in between so the race condition exists for sure. We could upload buildhub.json in the same time with checksums, but that requires some deps/payload overhead in beetmover, checksums-signing and beetmover-checksums.