Auto-sync /bin with gitian

[TheBlueMatt]

We probably should have done this a while ago, but it would be great if someone could write a script that could be run that would automatically download (from...?) new release binaries and verify them against N-of-M known gitian signers before placing them in the /bin/ folder (and maybe even updating the release notes). This would remove dependence on any single maintainer for the website-updating part of the release process.

[harding]

For the release notes, there are often subtle differences between github-flavored markdown and this site's Kramdown-markdown which need to be resolved with some small edits. If we really want to automate that, I can maybe describe the problems I've encountered over the years and we can try to fix them with sed, but it sounds a bit like a pain. We also currently use a Jekyll plugin to automatically link-ify PR numbers and commit ids, so that would need to be integrated.

So, although I'd love to see the release notes part automated, maybe that should be a stretch goal.

[TheBlueMatt]

Fair enough, at least the /bin uploading could be automated, even if the UI updating not so much.

[laanwj]

We probably should have done this a while ago, but it would be great if someone could write a script that could be run that would automatically download (from...?)

Another idea would be not to download it from somewhere else, but spin up a build VM on the server itself. If it matches the gitian sigs, make the files available. This seems as close to a hermetically sealed distribution server as you can get.

(I'm not entirely sure how to handle SHA256SUMS.asc and the torrent in this case though—the torrent could be generated on the server as well, this is what bitcoin.org used to do. But if we still want to PGP-sign the hashes it's seems preferable to do this offline, or at least separate from the distribution server)

[harding]

I'm not an expert, but IIUC, magnet URIs are just the hash of the underlying torrent. If the torrent file was generated as part of the gitian/guix build process, then the magnet URI could be something attested to by the gitian signers. In that case, all the script would need to do is periodically git pull the gitian repo, notice when n of k trusted signers had committed to the same magnet URI, then use a torrent client to grab those files. That requires trusting git, GPG, and the torrent client not to have remotely exploitable bugs, which I think is a bit more than the current risk of trusting openssh, but it doesn't seem entirely crazy to me.

[laanwj]

then the magnet URI could be something attested to by the gitian signers

I like that idea. But mind that the torrent includes SHA256SUMS.asc with PGP-signed hashes of all the distributed files. I'm not sure how that could be included in the gitian process.

Mind that I really already want to move away from using my key there, but bitcoin/bitcoin#20977 doesn't seem obvious.

[harding]

Maybe the gitian signing could be done in two passes:

1. Build the binaries, attest to all the resultant files in a PR (the entire current process)
2. After @laanwj has had a chance to create SHA256SUMS.asc, gitian signers can each run a simple shell script that downloads SHA256SUMS.asc and the detached sigs for macOS, checks that all the checksums are correct and the signature is valid for the expected key, and then attests to the torrent hash in a new PR.

Although we want to make gitian/guix signing as easy as possible, I don't think running an additional 50 line shell script and opening an additional PR would add much of a burden. Using multiple passes may also help with https://github.com/bitcoin/bitcoin/issues/20977 once a reasonable procedure has been worked out (e.g. threshold schemes may require two passes anyway: a non-interactive build step followed by an interactive signing step).