Publish additional metadata

[daewok]

This looked like the best repo for this, but let me know if another one would be better.

There is a lot of information available to quicklisp-controller that doesn't end up making its way into the published dist (like source repository information and system description). Would you be amenable to publishing this information even though the Quicklisp client doesn't currently use it?

If so, I'd be happy to work on that by documenting the current QL dist structure (unless this exists somewhere already), merging it into either this repo or a standalone documentation repo, and then proposing the extensions I have in mind via doc updates before modifying the controller to generate it.

[quicklisp]

Here are some quick thoughts:

• in quicklisp-controller, refuse to create a mock dist if the quicklisp-projects repo isn't clean

• in quicklisp-controller, produce a CDB file keyed on project name with a value of the source.txt first line, make it part of the mock dist

• in quicklisp-dist-publish, merge CDBs from existing and mock dist appropriately before uploading

• in quicklisp-client, add functions to fetch upstream-source.cdb on demand, and add GF UPSTREAM-SOURCE that returns string from CDB file given a ql-dist:system or ql-dist:release

• in quicklisp-client, if error on fetching upstream-source.cdb, signal some appropriate (continuable?) error; this will likely be common for dists in the wild

• in some one-off code, work out nice way to create upstream-source.cdb from tags in quicklisp-projects repo to older dists

My goal has always been to provide better provenance info. Specifically, for each upstream-source type, provide enough info to recreate the archive provided in the dist. So things like the repo URL, commit id, fetch date/time (for HTTPS sources), fetched headers, etc. Simply providing the source.txt data isn't really enough to fully recreate things in a reliable way. But it's a useful first step.

On Tue, Nov 17, 2020 at 11:34 AM daewok [email protected] wrote:

This looked like the best repo for this, but let me know if another one would be better.

There is a lot of information available to quicklisp-controller that doesn't end up making its way into the published dist (like source repository information and system description). Would you be amenable to publishing this information even though the Quicklisp client doesn't currently use it?

If so, I'd be happy to work on that by documenting the current QL dist structure (unless this exists somewhere already), merging it into either this repo or a standalone documentation repo, and then proposing the extensions I have in mind via doc updates before modifying the controller to generate it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/quicklisp/quicklisp-controller/issues/18, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACPNLJCJNLJ7MTCKLYVAQLSQKQ2RANCNFSM4TYZYNCA .

[daewok]

That does sound like a good first step. I'm curious though, what's the reasoning behind distributing upstream-source.cdb directly instead of building the cdb client side like systems.cdb and releases.cdb are?

A big desire of mine is to have an index of VCS info for each project. Including source.txt gets us most of the way there, but won't work for projects where source.txt grabs a release tarball instead of a git repo (like CFFI). So I was thinking of including a lot of the metadata from the system file itself, similar to what #14 was trying to accomplish. Alternatively, quicklisp-projects could contain something like a source-control.txt file, but I like the story of making all ASDF metadata available better (it also means projects could be encouraged to provide this metadata).