Storing the index

[tchoutri]

We have two possible courses of action here regarding the package index:

1. We keep using the append-only tarball that Hackage uses. Historical solution but well-understood. Not used much outside of Haskell
2. We switch to a git-based storage. Massively used by a lot of other PMs. Ease of mirroring as well? At least the storage handles the audit log.

• https://github.com/NixOS/nixpkgs/
• https://github.com/rust-lang/crates.io-index

[shadowcat-mst]

CPAN provides a snapshot file (02packages.details.txt.gz) downloadable from a known location, and also keeps the non-gz version of that file in a git repository so one can spelunk history.

Both are definitely valuable - as is a web service for querying said index (ala https://cpanmeta.grinnz.com/packages) but git as the primary storage for the index metadata is IMO obviously correct in 2021.

This is not entirely true for -data- rather than -metadata- since it's essential to have a (human mediated) means to completely delete a tarball if it accidentally contains proprietary information or API keys or something like that but for the pure metadata, sure, stick it in git.

(to anybody who might say "well, it's out there anyway at that point" my answer would be that, yes, I know, but refusing to delete such an upload is a good way to have corporate lawyers ban uploading anything ever, and whether that's sensible or not, I don't want to punish somebody for trying to contribute to open source and I don't want to accidentally cause entire companies to ban trying to do so any more than they already do)

Any other formats you might want therefrom should be produced using the moral equivalent of cron jobs.

[tchoutri]

Awesome, thank you very much!