x/tools, x/build: create a relui workflow for gopls release automation

[dle8]

This issue tracks the relui workflow to automate gopls release namely:

• [ ] Create an issue to track the release
• [ ] Creating a new release branch
• [ ] CL to update the release branch from master
• [ ] CL to update the debug version
• [ ] CL to update gopls/go.mod to require the correct x/tools
• [ ] Tag the prerelease
• [ ] Generate the release note
• [ ] Create a new release in github
• [ ] Tag a final release
• [ ] Update dependencies for x/tools and x/tools/gopls

CC @findleyr

[gopherbot]

Change https://go.dev/cl/462517 mentions this issue: internal/task: get the next version that would be tagged

[gopherbot]

Change https://go.dev/cl/462815 mentions this issue: gopls/release: remove unused functionality from release script

[gopherbot]

Change https://go.dev/cl/601236 mentions this issue: internal/task, cmd/relui: add definition for releasing go pls

[gopherbot]

Change https://go.dev/cl/601655 mentions this issue: internal/task, gerrit/gerrit: create a new branch if minor version

[h9jiang]

In the next change, we will add a step to create a branch for a minor version (X.Y.0) release.

In my local run, the release flow failed because of permission denied issue when trying to create the next minor release branch gopls-release-branch.0.17. (WAI) The permission issue should not happen in prod.

[h9jiang]

In the next change, we will add a step to change the codereview.cfg under the root dir of x/tools repo, and create a CL based on the change. The flow will take reviewers as optional parameters.

In my local run, I short circuit the previous steps (i.e. version validation, branch creation). Provide input parameter version = v0.1.0. The codereview.cfg in release branch gopls-release-branch.0.1 is not in desired state yet.

So the flow will create a CL to update codereview.cfg accordingly. ( Reconcile it to desired state. :D )

[gopherbot]

Change https://go.dev/cl/601241 mentions this issue: internal/task: update codereview.cfg and create a CL for review

[h9jiang]

In the next change, we will add a step to update the x/tools dependency in the gopls sub module, and create a CL based on the change.

The way of updating the dependency:

• Drop the replace directive of x/tools so the head of release branch will not point to its parent.
• Pin the x/tools to the head of the current release branch.

In my local run, I short circuit the previous steps (i.e. branch creation). Provide input parameter version = v0.16.2. The go.mod & go.sum in release branch gopls-release-branch.0.16 will be updated to the head of the release branch.

The CL created update the go.mod

- golang.org/x/tools v0.22.1-0.20240628205440-9c895dd76b34
+ golang.org/x/tools v0.22.1-0.20240715174654-66fa8e7351ea

And the git log -1

commit 66fa8e7351eaa1909b6afcaa95e7adb4e08b1714 (HEAD -> gopls-release-branch.0.16, origin/gopls-release-branch.0.16)
Author: Rob Findley <[email protected]>
Date:   Fri Jul 12 21:56:01 2024 +0000

    [gopls-release-branch.0.16] gopls: update x/telemetry dependency
    
    Update x/telemetry to pick up recent bug fixes (notably
    golang/go#68311).
    
    Change-Id: Ic95d54fc43872ab5c3548cf47134fbaea27e1487
    Reviewed-on: https://go-review.googlesource.com/c/tools/+/598115
    Reviewed-by: Dmitri Shuralyov <[email protected]>
    Reviewed-by: Dmitri Shuralyov <[email protected]>
    LUCI-TryBot-Result: Go LUCI <[email protected]>
    (cherry picked from commit 6f4e2a811e8b599d25464bc036a8abda0088d4ed)
    Reviewed-on: https://go-review.googlesource.com/c/tools/+/598056
    Auto-Submit: Robert Findley <[email protected]>

The CL change the version of x/tools in gopls submodule to the head of this release branch.

[gopherbot]

Change https://go.dev/cl/602516 mentions this issue: internal/task: update x/tools dependency and create a CL for review

[h9jiang]

In the next change, we will add a step to verify we can successfully install gopls from the head of the branch using the go installgolang.org/x/tools/gopls@{BRANCH_NAME}`.

Update: The gopls verification will download the gopls using the commit from the previous merged CL. (If the previous step does not merge any CL, using the commit from head of release branch). Run a simple smoke test by gopls references

[gopherbot]

Change https://go.dev/cl/603036 mentions this issue: internal/task: verify go install after x/tools dependency update

[h9jiang]

In the next change, we will add a step to tag the pre-release for a given release.

In my local run, I tried to release a new gopls version 0.16.2. (This is the next patch version so there is no tag related to this release). The flow realize he should tag this release branch's (gopls-release-branch.0.16) head commit gopls/v0.16.2-pre.1 (the pre-release version start from 1).

My personal credential do not have permission to tag this repo. So I get 403 permission denied error.

(Just like steps above, I did short-circuit a few steps to make this happen)

[gopherbot]

Change https://go.dev/cl/602520 mentions this issue: internal/task: add pre-release tag to release branch's head

[gopherbot]

Change https://go.dev/cl/603875 mentions this issue: internal/task: internal/task: refactor milestone methods

[h9jiang]

Next change, the release automation will find the release milestone and create a release tracking issue. All the CL created for this release should reference this issue (part of the commit message).

Happy case 1: release milestone exist, release issue does not exist. A new issue should be created, CL should have this issue number in commit message.

Happy case 2: release milestone exist, release issue exist. All the CL should refer to this pre-existing issue.

Failure case: the milestone is closed. The flow should error out.

@findleyr @dmitshur

[gopherbot]

Change https://go.dev/cl/603876 mentions this issue: internal/task: create releasing issue under release milestone

[h9jiang]

Next change, the release automation will re-run a verification after tag the pre-release version. So the workflow will verify the gopls installation twice. From the high level, the flow looks like this:

1. Verify installation using go install gopls@COMMIT (commit is head of release branch)
2. Tag the head of the release branch with gopls.vX.Y.Z-pre.V
3. Verify installation using go install [email protected] (this is tag we just created)

In my local relui, a lot of steps are being short circuited.

[gopherbot]

Change https://go.dev/cl/604295 mentions this issue: internal/task: verify the gopls installation with pre-release tag

[gopherbot]

Change https://go.dev/cl/604117 mentions this issue: internal/task: rename release gopls to prerelease

[h9jiang]

Next change, we will rename the current flow to pre-release flow. The pre-release flow is responsible for preparing a pre-release candidate but the release flow is responsible for release a prepared pre-release candidate.

So a new release will be create to handle the actual release job. This flow will accept a full semantic version string as input in form of vX.Y.Z-pre.V. This flow will first validate the input version is semantic correct and logically correct.

• Is the latest pre-release for its Major.Minor.Patch.
• Has a corresponding tag in the x/tools repo.
• The tag points to the head commit of the release branch.
• The corresponding release tag does not exist in this repo.

Error case, I provide v0.16.0-pre.3 as input parameter. This flow should error out. Because head of the release branch is way ahead of this tag.

Error case, I provide the v0.14.2-pre.1 as input parameter, this flow should error out because this specific release is already done. There is a release tag exist in this repo.

Unfortunately, the current state of x/tools do not have any happy case I can demonstrate. (no ongoing release like 0.16.3-pre.1). All of the release have both the pre-release tag and release tag.

[gopherbot]

Change https://go.dev/cl/604895 mentions this issue: internal/task: add definition for releasing gopls

[h9jiang]

Next change: as an improvement, we will replace current approach updating the x/tools dependency using branch with using commit.

This is to avoid any possibility where the module proxy provide some outdated info about the branch.

Normally, this won't be a concern, but in the gopls release, we trigger this go get command immediately after the previous CL merged. If we use go get .... $BRANCH, it have a higher chance to provide outdated info.

My local relui try to execute the command using commit, and create a auto submit CL.

go get golang.org/x/tools@66fa8e7351eaa1909b6afcaa95e7adb4e08b1714

[gopherbot]

Change https://go.dev/cl/605035 mentions this issue: internal/task: replace go get using branch name with commit

[h9jiang]

Next change: add a gopls release tag to x/tools release branch.

The coordinator will provide a semantic version as input to gopls release flow. This input version should follow the semantic version schema and the patch version should not be empty.

The flow will try to find the commit of this pre-release version and tag it with the official release tag.

E.g. input is v0.16.0-pre.2, the flow will find the commit where this tag is pointing to 6c3e23fca26b0bf0433f02b12aec71e3e8d2405b and tag this commit with release tag v0.16.0.

In my local relui below, the task failed because the tag already exist and I do not permission to tag this repo. :D

[gopherbot]

Change https://go.dev/cl/604999 mentions this issue: internal/task: add a release tag based on the input pre-release version

[gopherbot]

Change https://go.dev/cl/605656 mentions this issue: internal/task: add E2E test to pre-release gopls flow

[h9jiang]

Next change: fix a bug where the cloud run is being triggered against the master branch but the script is being triggered against the release branch.

[gopherbot]

Change https://go.dev/cl/605979 mentions this issue: internal/task: run script against the release branch

[h9jiang]

Next change: for gopls release, after add the release tag to the x/tools, the last step is to update the dependency for master branch. This step only need to be triggered for minor version release.

This step is suppose upgrade dependency for x/tool module and it's gopls submodule. The dependency upgrade is independent from each other, so two CL will be created in parallel.

For local relui test, I short circuit the version validation and release tag. (I just want to test if the dependency works or not).

The flow execution

The dependency upgrade for root dir was successful:

The dependency upgrade for gopls sub was failing:

The failure is reproducible locally.

/codebase/tools/gopls/ [master] go get -u all
go: downloading github.com/jba/templatecheck v0.7.0
go: downloading github.com/google/safehtml v0.1.0
go: downloading honnef.co/go/tools v0.5.1
go: downloading golang.org/x/vuln v1.1.3
go: downloading mvdan.cc/xurls v1.1.0
go: downloading golang.org/x/telemetry v0.0.0-20240815150606-0693e6240b9b
go: downloading golang.org/x/sys v0.24.0
go: downloading golang.org/x/exp v0.0.0-20240808152545-0cdaa3abc0fa
go: downloading github.com/BurntSushi/toml v1.4.0
go: downloading golang.org/x/exp/typeparams v0.0.0-20240808152545-0cdaa3abc0fa
go: golang.org/x/[email protected] (matching all@upgrade) requires golang.org/x/[email protected], not golang.org/x/[email protected] (matching all@upgrade)
go: honnef.co/go/[email protected] (matching all@upgrade) requires github.com/BurntSushi/[email protected], not github.com/BurntSushi/[email protected] (matching all@upgrade)
/codebase/tools/gopls/ [master] echo $?                         
1

But failure is not within the scope of this CL. So it's working as intended. The purpose of this CL is to upgrade dependency.

[gopherbot]

Change https://go.dev/cl/606336 mentions this issue: internal/task: update dependency in master branch after minor release