Introduce a Go workspace

[ivanvc]

What would you like to be added?

After discussing this in last week's community meeting and based on feedback from the Go team (https://github.com/golang/go/issues/68254) due to the vulnerability (GHSA-5x4g-q5rc-36jp / golang/vulndb#2952), there are several benefits to introducing a Go workspace in the project, and one of the biggest motivations is to simplify the test scripts.

I have a branch with the go.workspace (diff). It requires changes in the build scripts and the test libraries. It still doesn't work, but there's some progress.

I mostly based it on how kubernetes/kubernetes defines the Go workspace. However, from golang/go#68254 and the motivation to have govulncheck spot vulnerabilities within our modules, their suggestion is to remove replaces pointing to local code in go.mods, but k/k still has these replaces.

I wanted to open up the discussion to get feedback and/or implementation ideas.

Progress track

• [x] Workspace activation: #19423
• [ ] verify targets
  • [x] lint: #20788
  • [x] gofmt: #20814
  • [x] bom: #20829
  • [x] dep: #20848
  • [x] ~shellcheck~
  • [x] goword: #20815
    • [x] We can remove go_srcs_in_module after merging #20815 and #20812
  • [x] govet: #20810
  • [x] license-header: #20812 / #20818
  • [x] mod-tidy: #20828
  • [x] ~shellws~
  • [x] ~proto-annotations~
  • [x] ~genproto~
  • [x] ~yamllint~
  • [x] govet-shadow: #20810
  • [x] ~markdown-marker~
  • [ ] go-versions: Somewhat blocked by https://github.com/golang/go/issues/75941
  • [x] gomodguard: #20849
• [x] fix targets
  • [x] bom: #20829
  • [x] lint: #20788
  • [x] yamllint: #20850 / #20857
  • [x] ~sync-toolchain-directive~
  • [x] scripts/fix.sh
    • [x] mod-tidy: #20851
    • [x] go fmt: #20858
    • [x] bashws: #20860
• [ ] test targets
  • [x] unit: #20872
  • [x] integration: #20909
  • [x] e2e: #20925
  • [x] grpcproxy-integration: #20954
  • [x] grpcproxy-e2e: #20954
  • [ ] e2e-release
  • [ ] release
  • [x] robustness: #20962
  • [ ] coverage
• [ ] build targets

Why is this needed?

To improve the code quality and to spot vulnerabilities in our code firsthand.

[ahrtr]

High level I support this. I raised this about 3 years ago https://github.com/etcd-io/etcd/issues/13478, but go workspace wasn't widely used at that time. It should good timing now.

[ivanvc]

This was discussed during the community meeting. I'll draft a document to discuss options for implementing this change.

[ivanvc]

I'm sorry it took me a very long time to draft the previously mentioned document. Here's the draft: https://docs.google.com/document/d/1Vpb0SosYT05YsBsgfJFzOAbSgitd1BcFOQzjEARaQyM. @ahrtr, PTAL and let me know your thoughts.

[ahrtr]

Based on the discussion in our yesterday's community meeting, we can create a separate branch (i.e workspace) for the go workspace change, the rough steps are,

• disable dependabot, so that we won't update the go.mod and go.sum files during the development of the feature on main branch.
• freeze the change on scripts on main branch. (Please anyone chime in if you have any related PR that you want to get merged before this point)
• create a new dev branch (i.e workspace)
• multiple contributors work on the feature until it's done and all workflows are green. (hopefully it won't take too long, i.e < 8 weeks)
• merge the branch back to main
• un-freeeze the script change on main
• recover the dependabot.

cc @ivanvc @jmhbnz @serathius

[jmhbnz]

Based on the discussion in our yesterday's community meeting, we can create a separate branch (i.e workspace) for the go workspace change

I have some etcd meeting videos to edit and upload from last couple weeks, will watch recording then comment but at face value it seems reasonable provided everyone is on board.

[ivanvc]

I'll start by breaking this up into smaller tasks. After that, I'll try to recruit some people to help. Then, we'll be able to proceed with the branch and pause dependabot.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[serathius]

What we are missing to enable it on main branch?

[ivanvc]

After our discussion, I attempted to create the workspace without the tools module(s), but ultimately, it failed. I couldn't find an easy way to make it work this way.

I believe we agreed based on @ahrtr's suggestion to consider it/merge after 3.6.0 to avoid having a significant discrepancy between the main and release-3.6 branches.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[serathius]

Can't wait for this to happen!

[ivanvc]

Sorry, I was busy the last few weeks. This is next on my to-do list :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.