new release process / CI actions / checks

[sam0x17]

trafficstars

We want to move to a more formal release process characterized by the following points:

• all PRs merge into main
• PRs don't get merged until they are finney-ready
• labels will be used to indicate that a PR is on-devnet, devnet-pass, on-testnet, and testnet-pass, respectively, and having both the devnet-pass and testnet-pass labels will be required by the CI to merge into main.
• any issues uncovered during testing can then be discussed and tracked on the PR, reducing issue/PR bloat and making it easy to see the full life-cycle of a feature from development to deployment on finney
• we will cut releases off of main using the github releases feature, so every once in a while we cut a new release and deploy everything new in main to finney, and then that gets a semver version number + tag
• also new github actions to run our (now manual) deploy process, these will include a built in check that the spec version has been bumped to a value higher than the net we are currently deploying to. If this criterion is not met, the deploy will abort, which will prevent issues like the recent testnet forking/bricking

all work in #346

AC:

• [x] CI check system for devnet / testnet stages (possibly using labels)
• [ ] github actions for deploys
• [ ] set up releases
• [ ] PRs default to main
• [ ] update docs / instructions / PR templates
• [ ] test that non-team-members can't fake the checklist items

[gztensor]

PRs don't get merged until they are finney-ready

It needs a formal definition, i.e. what tests need to pass. E.g. does it imply testing on a full live data in Finney?

we will cut releases off of main using the github releases feature, so every once in a while we cut a new release and deploy everything new in main to finney, and then that gets a semver version number + tag

What if the main branch goes forward from the commit that we want to release? How do we update version in the release? How do we apply hot fixes if we don't want further commits (already present in main branch) to be included in this release? Does it mean merging of PRs is blocked until the release is finished? What happens to the on-going feature branches if they need to merge upstream commits that are not included in this release during the release process?

[sam0x17]

@gztensor

It needs a formal definition, i.e. what tests need to pass. E.g. does it imply testing on a full live data in Finney?

So right now this is defined based on the label rules which means it has to have the devnet-pass and testnet-pass labels which means:

1. has been deployed to devnet successfully
2. has been manually tested on devnet
3. has been deployed to testnet successfully
4. has been manually tested on testnet

I can also add other required labels if necessary

What if the main branch goes forward from the commit that we want to release?

This will regularly happen, the tip of main will normally be ahead of our current release which will usually be a few commits back in main. Releases will be tagged and uploaded as a GH release using an action automation

How do we update version in the release?

I'll be doing this manually with each release. One thing that might help this process is if we add labels or something to indicate on PRs what level of semver change they necessitate, so I'll definitely add something about this to the PR templates and/or docs

How do we apply hot fixes if we don't want further commits (already present in main branch) to be included in this release?

When you cut a release it can be based on any tag you like, so that's how we would do that. If there is a hotfix we merge it into main and cut a release. Everything already in main that hasn't been deployed to finney will already be, by definition, ready to go on finney, so that will be fine. With the old approach this is more of a problem because we have things in the development branch that aren't yet tested, but with this approach everything that merges into main is actually ready to go on finney, so we can hotfix off of that whenever we want without fear.

[sam0x17]

related Q: how do we feel about semver with respect to this stuff -- what sorts of things constitute a patch, minor, and major version change when it comes to subtensor?

[gztensor]

Thanks for details, @sam0x17 ! I have just one concern that this pipeline may become a bottleneck if we have too many pending PRs waiting for approval. If features depend on each other, it will be tempting to cross-merge without waiting until the PR is accepted in the main branch. Would be great if we have some rules about cross-merging to prevent mess.

[sam0x17]

Thanks for details, @sam0x17 ! I have just one concern that this pipeline may become a bottleneck if we have too many pending PRs waiting for approval. If features depend on each other, it will be tempting to cross-merge without waiting until the PR is accepted in the main branch. Would be great if we have some rules about cross-merging to prevent mess.

I agree this is a potential issue, but so is the churn created by everyone being rebased off of a development branch that then needs changes because not everything is checked.

My hope is that we can do more frequent releases and that that will be the answer to this concern. Right now stuff can't move forward until all the other stuff in its "cohort" is tested, this would let things move more independently of each other