cmd/go: mod tidy reports toolchain not available with 'go 1.21'

[matthewhughes-uw]

Note: I think this is a question of: is "go 1.21" a valid go directive? since there was no 1.21 release, only 1.21.0, perhaps not

What version of Go are you using (go version)?

$ go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

Yes, reproduced on Go 1.21.0

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/proj/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build435086183=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Given a basic go.mod

$ cat go.mod 
module example.com/foo

go 1.21

Then run:

$ GOTOOLCHAIN="go1.20+auto" go mod tidy
go: downloading go1.21 (linux/amd64)
go: download go1.21 for linux/amd64: toolchain not available

What did you expect to see?

go mod tidy runs successfully. I guess I expect it would pick go1.21.0 as the toolchain if no toolchain is available as 1.21

What did you see instead?

The error output above with a non-zero exit code

More details

The issue was seen when running dependabot on a repo with a go 1.21 directive in go.mod, the GOTOOLCHAIN above was taken from their usage https://github.com/dependabot/dependabot-core/blob/08ac25ebd773cede0c00be9a98e5bb03b680870b/go_modules/Dockerfile#L34

Running the above command with GODEBUG=http2debug=1 I see it:

• requests go.dev/dl/mod/golang.org/toolchain/@v/v0.0.1-go1.21.linux-amd64.zip which returns Location: https://go.dev/dl/mod/golang.org/toolchain/@v/v0.0.1-go1.21.linux-amd64.zip
• requesting that URL then returns location: https://dl.google.com/go/v0.0.1-go1.21.linux-amd64.zip
• requesting that URL gives a 404

Updating the directive to: go 1.21.0 will permit things to run fine.

[seankhliao]

how did you end up with go 1.21 as a directive?

[matthewhughes-uw]

how did you end up with go 1.21 as a directive?

I can't remember exactly, but I think it was rather manual like: go1.21.0 mod edit -go=1.21

[laboger]

I am seeing this problem on linux/ppc64le. I have built a toolchain from the latest go1.21 branch and set up my PATH to use it. When I cd'ed to cmd/compile/internal/ssa in a toolchain from master that changes the behavior to what is shown above. I found it when I tried to do 'go generate' but I get the same behavior from just doing 'go version' from within that directory. If I cd back to another directory then it provides the correct go version.

$ go version
go version go1.21.0 linux/ppc64le
$ cd ~/golang/plain/go/src/cmd/compile/internal/ssa
$ pwd
/home/boger/golang/plain/go/src/cmd/compile/internal/ssa
$ go version
go: downloading go1.22 (linux/ppc64le)
go: download go1.22 for linux/ppc64le: toolchain not available
$ cd ~/gotests
$ go version
go version go1.21.0 linux/ppc64le

[bcmills]

@matthewhughes-uw, this is as expected: go 1.21 was a development version of the language, for which there is no downloadable release (because it is not a specific version). The release version would be go 1.21.0.

In general if you want to specify a development version in the go line, you must also give a concrete toolchain version. So either of these should work:

go 1.21.0

or

go 1.21
toolchain go1.21.0

[bcmills]

@laboger, the problem you are seeing is similar. go1.21.0 cannot upgrade automatically to a toolchain that supports go 1.22, because no such toolchain has been released. In order to compile a package in a go 1.22 module, you need to use a development build of the toolchain that supports that language version.

In particular, if you are working within $GOROOT/src you should be using exactly the toolchain built by make.bash within that GOROOT — you may need to re-run make.bash and/or check your $PATH. 😅

[bcmills]

As far as I can tell this is all working as designed. @matthewhughes-uw, are there specific documentation changes that would have helped you solve or avoid this problem?

[matthewhughes-uw]

As far as I can tell this is all working as designed

:+1:

@matthewhughes-uw, are there specific documentation changes that would have helped you solve or avoid this problem?

I think the main issue was my assumption that, previously I could just specify a language version (per https://go.dev/doc/toolchain#version) i.e. go 1.X in my go.mod e.g. go 1.20 but with Go 1.21 I can no longer do this. The docs say:

The version must be a valid Go release version, such as 1.9, 1.14, or 1.21rc1

and within the linked page

The syntax ‘1.N’ is called a “language version”. It denotes the overall family of Go releases implementing that version of the Go language and standard library.

so I guess it wasn't clear to me that a "language version" != a "Go release version" (though it was true for every language version before Go 1.21)

In general if you want to specify a development version in the go line, you must also give a concrete toolchain version. So either of these should work:

:thinking: Maybe it's worth documenting something like "If you want to specify a 'language version' in go.mod you should (must? But again, only true for Go 1.21 at the moment) also specify a toolchain version"?

[prymitive]

I think the main issue was my assumption that, previously I could just specify a language version (per https://go.dev/doc/toolchain#version) i.e. go 1.X in my go.mod e.g. go 1.20 but with Go 1.21 I can no longer do this.

To add to this: what really happened here that causes a bit of confusion is that the value of go keyword in go.mod changed it's format.

Before 1.21 release X.Y.Z value (1.20.1) would be invalid so you couldn't put it there.

With 1.21 release you most likely want to put X.Y.Z there since most user would want to put there a real working version, not a dev release. To add to confusion X.Y value technically works, but it may or may not depending on what was released.

So the way I see it we went from never put .0 to always put .0 (0 being point release number).

Having now also toolchain (which from docs sounds like it's optional) just means that people need to wrap their heads around more complicated set of options to set, as per example from https://github.com/golang/go/issues/62278#issuecomment-1693538776.

[bcmills]

After some further discussion with @hyangah (#65580), I think it should be possible to fix this, specifically for commands that already modify the go.mod file (such as go mod tidy and go get) if there is a release available above the version recorded in the go.mod file.

[jamietanna]

Related: I'm seeing this for the first time with Go 1.22:

/app # env GOTOOLCHAIN=auto go build                                                                          
go: downloading go1.22 (linux/amd64)                   
go: download go1.22 for linux/amd64: toolchain not available    

As mentioned, the following diff fixes this:

diff --git go.mod go.mod
index 431a4ba..1ed7b4e 100644
--- go.mod
+++ go.mod
@@ -1,6 +1,6 @@
 module dmd.tanna.dev
 
-go 1.22
+go 1.22.0

[amnonbc]

This may be working as designed, but it does violate the principle of least astonishment, and is biting multiple people upgrading to a new Go version. As others have noted, before 1.21, the go.mod specified a two digit version number. When upgrading to 1.22, a two digit version fails when building with an older tools, but succeeds for people with 1.22 already installed.

[LINKIWI]

It's not clear to me from the comments in this issue what the correct user-facing fix is.

My local Go version is:

$ go version
go version go1.21.7 linux/amd64

I edited my module's go.mod as follows:

module ...

go 1.22.0

...

But this still fails in the same way:

$ go mod tidy -v
go: downloading go1.22.0 (linux/amd64)
go: download go1.22.0 for linux/amd64: toolchain not available

[matthewhughes934]

It's not clear to me from the comments in this issue what the correct user-facing fix is.

My local Go version is:

$ go version
go version go1.21.7 linux/amd64

I edited my module's go.mod as follows:

module ...

go 1.22.0

...

But this still fails in the same way:

$ go mod tidy -v
go: downloading go1.22.0 (linux/amd64)
go: download go1.22.0 for linux/amd64: toolchain not available

Can you share the entire go.mod file? Running the golang:1.21.7 I see a succesful run via:

$ go mod init proj
$ go mod edit -go=1.22.0
$ cat go.mod 
module proj

go 1.22.0
$ env --unset GOTOOLCHAIN go mod tidy # in the image GOTOOLCHAIN was set to "local" so unset it so a download is attempted
go: downloading go1.22.0 (linux/amd64)
go: warning: "all" matched no packages

[adonovan]

See also https://github.com/golang/go/issues/66175, in which I request a better error message about a missing patch version suffix.

[eatmoreduck]

@matthewhughes-uw, this is as expected: go 1.21 was a development version of the language, for which there is no downloadable release (because it is not a specific version). The release version would be go 1.21.0.

In general if you want to specify a development version in the go line, you must also give a concrete toolchain version. So either of these should work:

go 1.21.0

or

go 1.21
toolchain go1.21.0

addd toolchain go1.21.0 works for me ! thanks

[bcmills]

(CC @matloob @samthanawalla)

[ldemailly]

~shouldn't go mod tidy add the toolchain if it's required~, I have the following problem:

installed go version/version of go in the PATH: 1.21.8 local directory go.mod with go 1.22

go version fails: https://github.com/golang/go/issues/65568#issuecomment-2010111943

and even trying to fix it, it still fails:

$ GOTOOLCHAIN=go1.22.1 go version
go: downloading go1.22.1 (darwin/arm64)
go version go1.22.1 darwin/arm64
$ GOTOOLCHAIN=go1.22.1 go mod tidy
$ git diff

(no change, no addition of toolchain) so back to:

$ go version
go: downloading go1.22 (darwin/arm64)
go: download go1.22 for darwin/arm64: toolchain not available

which anyway should really just mean use latest go1.22 instead of breaking

Edit: imo it's fine that go mod tidy doesn't change go 1.22 nor add unnecessary toolchain and the bug is really that go 1.22 just means, for go users, any release of go 1.22 so if needed, use the already downloaded or download the most recent one (go 1.22.1 as of this writing) but in any case do not fail like it does right now without at least saying the equivalent of "currently running go will try to use ..."

[amnonbc]

This issue has also baffled Alan Donovan, the guy who literally wrote the book on Go. https://github.com/golang/go/issues/66175 Further evidence that the current behaviour violates the principle of least astonishment.

[haruyama480]

Let me summarize this problem to share.

When it occurs

• go.mod specifies Go language version over 1.21 in go.mod/work without toolchain directive.
  • e.g. go 1.21
• And, go command needs to download a newer toolchain.
  • e.g. compiling modules having go 1.22 with older toolchain like go1.21.x

How it occurs

• The toolchain directive added in Go 1.21 refers the version of the go directive if toolchain omitted
• However, the language version is not valid as a toolchain name unlike go directive. https://tip.golang.org/doc/toolchain#name
• So, downloading toolchain fails

[haruyama480]

Suggestion

• Handle the problem case as an invalid configuration to make go mod tidy fail
• If possible, suggest to specify go 1.21.0 or toolchain 1.21.0 as @bcmills says
  • this is also current workaround

But, I'm not sure this is easy to implement.

Up to 1.21, users have been installing manually go when Go language updates. From 1.21, manual installation will be unnecessary at all thanks to awesome toolchain system, If users likely specify go1.21.0 without confusion(https://github.com/golang/go/issues/62278#issuecomment-1698829945)!

[haruyama480]

Oh, my suggestion seems to cause many scripts that is already running to fail. Improving some linters may be better because of less unintended failures.

[haruyama480]

I found another workaround.

Adding a following line to .bashrc (or equivalent) and updating it for next go release, forces go command to download valid new toolchain which satisfies go directive.

export GOTOOLCHAIN='go1.22.0+auto'

It only works in a local environment. But, it doesn't need changing code.

[matloob]

We're closing this in favor of issue https://github.com/golang/go/issues/66175. If we can tell the user to list go 1.21.0 in their go.mod instead of go 1.21, that will help people correct their go.mod files.

[hyangah]

This was closed in favor of #66175 that asks for improved error messages. I hoped this issue can allow use of the language version syntax (or the major.minor version syntax) to continue to work (instead of toolchain version syntax).

Inside google, users often end up using a modified go toolchain installed on their machines. That produces go.mod and go.work with the major.minor syntax (since the toolchain is not the stable, released version). If the code is published with the major.minor syntax, later external users will encounter this issue and have to modify the version in go.mod. That may be confusing.

Why couldn't the cmd/go pick up a reasonable tool version if it is able to suggest users to list go 1.21.0 when seeing go 1.21? If it's now a new best practice to specify the toolchain version in the go line, do we still need a separate toolchain line in go.mod?

cc @golang/tools-team

[matthewhughes-uw]

If it's now a new best practice to specify the toolchain version in the go line, do we still need a separate toolchain line in go.mod?

If I understand correctly these still serve separate purposes:

• go directive: minimum version of Go that is supported/works with the module, e.g. your code uses some part of the stdlib introduce in go1.22.0 so that is the version you should use here
• toolchain directive: minimum version of Go for working in the module, the only time I've used this is: Go 1.22 introduced coverage summaries for packages with no test files, so when developing my module (which builds for/supports go1.21.0+) I want to use 1.22.0+ for nicer test coverage, so I set go 1.21.0 and toolchain go1.22.0.

[ldemailly]

but why would a patch level be needed for minimum version one needs/support, by definition no language/library api change happens in patches

[matthewhughes-uw]

but why would a patch level be needed for minimum version one needs/support, by definition no language/library api change happens in patches

Bug fixes do though https://go.dev/blog/toolchain#forward says:

Forward compatibility refers to what happens when a Go toolchain attempts to build Go code intended for a newer version of Go. If my program depends on a module M and needs a bug fix added in M v1.2.3, I can add require M v1.2.3 to my go.mod, guaranteeing that my program won’t be compiled against older versions of M. But if my program requires a particular version of Go, there hasn’t been any way to express that: in particular, the go.mod go line did not express that.

so I guess a use case would be: there's a bug in some part of the go toolchain that:

• Doesn't impact how your module compiles/runs, but
• Does impact how you develop your module

[matloob]

After further consideration and discussion with @hyangah @rsc and @samthanawalla we've decided to change the behavior so that if the go.mod lists go 1.X and the current go version is less than that version, we'll attempt to download go 1.X.0 if it exists.

Many of our users are used to manually writing go 1.X in their go.mod files so it seems reasonable to allow that to continue to have a reasonable meaning. Though once this is fixed and released in 1.23 the fix will only be available to users with local toolchains of 1.23 and later.

[haruyama480]

@matloob

Thanks for the discussion!

I am wondering about a corner case. If this has already been discussed, please disregard.

Can we build a module having go 1.X with the toolchain go1.Xrc1? Note that the order of go versions is like 1.X < 1.Xrc1 < 1.X.0 when X>=21. If it works, there are difference between go 1.X and go 1.X.0. If it doesn't work, using release candidates requires go 1.Xrc1 or toolchain go1.Xrc1.

[Alonekkpthpttall]

Details