Tweak structure of the plugins repository

[itowlson]

The plugin installation code expects the manifest for the latest version of the plugin to be at zarjazz.json, with previous versions at [email protected], [email protected], etc.

A slightly vexing side effect of this is that in order for a plugin developer to update a plugin, they must move zarjazz.json to [email protected] and then put the new latest in its place. This feels at odds with what feels like a purely additive operation.

It also has the vexing side effect for reviewers of not seeing a clean diff with the new file. Instead they see the old version as a new file, with diffs for URLs, digests, etc. in the "current version" file. It is therefore tricky when reviewing to verify that the old version hasn't changed.

It would be good to tweak the plugin code to allow new versions to be added to the plugins repo in as purely additive a way as possible, with any diffs to existing files being minor and easily verifiable by eye. For example, we could version all JSON files (so that the new one comes in as [email protected] instead of zarjazz.json). And maybe we'd need a current.txt file to point to current but that would be easy to check, or maybe we could make the installer super duper smart enough to figure it out, or something.

Obviously there are ghastly compatibility implications for this.

Anyway throwing this out for consideration and discussion. (And doing it here because it's really about the Spin code, the plugins repo just follows what that dictates.)

[fibonacci1729]

+1; it's a small but annoying hassle to add new versions ATM. Your proposal of versioning all manifests and tracking latest (or intelligently deducing) makes sense to me!

[kate-goldenring]

One way this could work (and solve the compatibility issue would be) to have the latest duplicated in both [email protected] and zarjazz.json. Then, we can add CI to the plugins repo to automatically update the contents of zaejazz.json to always be the latest version of the plugin. Essentially, this could potentially be managed/aided with some automation in the plugins repo? Say i am updating the plugin to add [email protected] (prev version was [email protected]).

1. A maintainer of the plugins repo, creates a versioned json for each latest unversioned foo.json in each plugin directory
2. I create a PR to add [email protected] file to the plugin directory. Note, [email protected] and zarjazz.json are the same at this point.
3. A bot creates a commit to the PR to update zarjazz.json as well

[rajatjindal]

One way to do verification would be to have a CI process to validate the changes instead of manual verification.

Generally the changes to plugin repo are of two categories:

• a simple version bump
• some other changes (e.g. description change, or the download URL change etc)

For the first category of changes, a CI process can easily validate that

• the new [email protected] is same content as zarjazz.json@main
• the only diff between zarjazz.json@branch and zarjazz.json@main is the change in version numbers.

If both the above checks are OK, we automatically mark the PR as approved (or maybe just say it looks OK as a comment on the PR). As an extension we can auto-merge the PR as well. e.g. this is how kubectl plugins are released. (Disclaimer: I am the owner of krew-release-bot).

For the second category of changes, I think it will be rare to have "other" changes published to an existing manifest. I would even argue that maybe we should disallow that. (a published artifact should be immutable).

I had started implementing the code for first category of changes, but never finished it. If folks agree to this, I can try to get that polished out by next week and add that to our CI process.

[rajatjindal]

+1; it's a small but annoying hassle to add new versions ATM. Your proposal of versioning all manifests and tracking latest (or intelligently deducing) makes sense to me!

Hi @fibonacci1729 have you seen spin-plugin-releaser. I would be curious to hear your thoughts on if that can help address some of the pain points of the current release process.

[itowlson]

@rajatjindal I think I am misunderstanding your intent. The download URL almost always changes because most plugin URLs are GitHub releases. So the version is part of the URL. And the digest always changes because why would there be a new version if nothing had changed. So I'm not sure how we are ever in a "only the version number changes" situation - could you amplify?

[rajatjindal]

pseudo code for this would be something like this:

if ("lines changed".contains("anything-other-than-url-and-sha") {
   // NOT A VERSION CHANGE
   return false
}

// check following for all the platforms which are part of plugin yaml file
if (strings.ReplaceAll("old-version-url", "old-version", "new-version") == "new-version-url") {
  // this means only version number has changed in the url
  return true
}

// and an additional step to validate that the digest is as expected from the new url

Just FYI, this is the tool used for doing same for krew-index repo: https://github.com/ahmetb/krew-index-autoapprove. I am suggesting that we can tweak (or rewrite) a tool like that for our plugins index repo.

I am happy to chat about this in our next spin meeting if you think that could be useful.