Fix broken lists.linuxfoundation.org URLs

[kanzure]

This pull request updates lists.linuxfoundation.org urls to gnusha.org/url.

Background

Unfortunately, lists.linuxfoundation.org is no longer hosting the bitcoin-dev static email archives, nor any others for that matter. This has caused many links around the web to become broken.

Redirect web service

gnusha.org/url is a web service that redirects old style archive urls to a new location where the content can be viewed.

For example, a bitcoin-dev mailing list archive url like:

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html

using https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html

is redirected to:

https://gnusha.org/pi/bitcoindev/[email protected]/

The redirect mappings were generated from my archives and you can review the scripts I made to do that here: https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b

Alternative

Instead of using gnusha.org/url, another option would be to resolve all of the archive urls to their email Message-ID redirects for the bitcoin-dev mailing list and use those urls instead. In fact, either gnusha.org/pi/bitcoindev or https://mailing-list.bitcoindevs.xyz/bitcoindev/ could be used for this purpose.

Those links would look like:

https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/
https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/

If this is the preferred solution for the BIPs, then let me know how I can help. Take a look here.

Finally, one other alternative is to wait and do nothing. Maybe someone can convince Linux Foundation to put up the static HTML archives for posterity?

More background

For more information see:

• https://gnusha.org/url
• https://github.com/bitcoin/bitcoin/pull/29782#issuecomment-2460974096
• https://github.com/bitcoin/bitcoin/pull/31240
• https://github.com/bitcoinops/bitcoinops.github.io/pull/2018
• https://github.com/bitcointranscripts/bitcointranscripts/pull/566
• https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/
• https://x.com/kanzure/status/1853779672514826334

[jlopp]

tACK

[kanzure]

@jonatack, fixed that. thanks.

I am seeking feedback on gnusha.org/url/ vs resolving everything to urls like https://gnusha.org/pi/bitcoindev/[email protected]/

With the resolved urls, the IDs are more verbose-- instead of an integer ID for the message, it's the Message-ID from the actual email. Then in the future someone can use that Message-ID to find the email on a mirror, like https://mailing-list.bitcoindevs.xyz/bitcoindev/[email protected]/.

This seems to be the likely direction that bitcoin.git is going in. They only have four urls in the codebase.

[kanzure]

I'll elaborate on gnusha.org/url vs gnusha.org/pi/bitcoindev style urls.

It is nice and simple to use a find-and-replace with gnusha.org/url/ because anyone can easily repeat that exercise and verify the git diff before merging the pull request. However, the actual routing is not something that is verified by just checking that the git diff looks sane.

If I was to provide my mapping (I haven't uploaded this yet, but I did publish the scripts I used to make it), then someone could write a script and/or cross-verify that all of the replacements are valid according to the mapping, if we were to use the destination urls instead of the redirect service for this pull request. Then someone would be able to verify that kind of pull request.

There are other further issues though, like content malleability, wrong redirect link injection, service provider compromise... instead of referencing by Message-ID maybe in the future we can reference by email hash.... Just dreaming here.

[kanzure]

Instead of using gnusha.org/url, another option would be to resolve all of the archive urls to their email Message-ID redirects for the bitcoin-dev mailing list and use those urls instead. In fact, either gnusha.org/pi/bitcoindev or https://mailing-list.bitcoindevs.xyz/bitcoindev/ could be used for this purpose.

this is implemented here: https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py

LMK if that's the preference instead of the current version of this pull request (gnusha.org/url inserts).

[jonatack]

Thanks @kanzure. No strong opinion, but I think I mildly prefer the destination urls over the redirects.

[kanzure]

There would be no delay. Either will work now. See the gist.

[ajtowns]

The lists.linuxfoundation.org site seems to redirect automatically now:

$ curl -I https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-June/001511.html
HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-June/001511.html

(also applies to lightning-dev mails)

Is there still a good reason to churn a bunch of bip texts, rather than just restrict this to updating the mailing list location in bip 2?

[kanzure]

The lists.linuxfoundation.org site seems to redirect automatically now:

I haven't announced this yet. It's a strange situation because LF has not agreed to keep these redirects (perhaps they don't even know at the moment about these redirects), and LF is not running that server. Given the strangeness I haven't been sure what to say or announce precisely.

[murchandamus]

Hey @kanzure, do you have an update on this?

[kanzure]

My last comment still stands. No updates from that.