Automate release testing

[waynew]

In order to actually focus on automating the release tasks, we need an issue for it.

This issue is to track the automation of part of the release process

The high level design is that we should have a single command that can be executed, and the end result is that we should have a pass/fail, and especially for pass we MUST have the accompanying release artifact. We should follow the release process, but basically this looks like:

• checkout the most recent salt/main code (maybe create a worktree for it?? Oh! No - can checkout the contents of the repo in another directory without actually tracking it with git. I forget what this is called.)
• build the wheel aka test artifact
• create a virtualenv and install the test artifact to it, along with test/doc requirements
• build the docs - errors in docs build mean the build has failed
• run the tests against the build/installed artifact - failures in tests mean the build was failed
• at this point if there are any failures, we should halt and bail out
• update the version.py file to the desired change, add & commit
• re-run the build/install/test workflow with this final release candidate -- I suppose in theory we could bump the version prior to running the build/test cycle? However, building from the latest main branch ensures that failures are completely unrelated :thinking:
• display pass/fail, and path of the release artifact that was tested

---

I think we should wait on automating creating the tag on pass - but that's something that we could do once we have towncrier for building the changelog, since the annotation on the tag should just be the contents of the changelog.

This means that we could do something like:

$ create-release 2021-11-10
...
PASS - dist/saltext.vmware-21.11.10-py3-none-any.whl
$ git tag -a 21.11.10
$ twine upload dist/saltext.vmware-21.11.10-py3-none-any.whl
$ git push salt 21.11.10

Followed by creating a release on GitHub.