cmd/go: `go mod tidy` ignores `go.work` file

[bozaro]

What version of Go are you using (go version)?

$ go version
go version go1.18beta1 linux/amd64

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bozaro/.cache/go-build"
GOENV="/home/bozaro/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/bozaro/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/bozaro/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.18/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18beta1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/bozaro/github/go-work-play/tools/go.mod"
GOWORK="/home/bozaro/github/go-work-play/go.work"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4292218461=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Minimal reproducing repository: https://github.com/bozaro/go-work-play/tree/go-mod-tidy

Full script:

$ git clone https://github.com/bozaro/go-work-play.git -b go-mod-tidy
$ cd go-work-play/tools
$ go mod tidy
go: finding module for package github.com/bozaro/go/go-work-play/shared/foo
github.com/bozaro/go/go-work-play/tools/hello imports
	github.com/bozaro/go/go-work-play/shared/foo: module github.com/bozaro/go@latest found (v0.0.0-20200925035954-2333c6299f34), but does not contain package github.com/bozaro/go/go-work-play/shared/foo

What did you expect to see?

I expect go.mod and go.sum updated with current working copy state.

What did you see instead?

I see that go mod tidy try to get modules for shared go modules from repository ignoring go.work content.

[bcmills]

Tidiness is a property of an individual module, not a workspace: if a module is tidy, then a downstream consumer of the module knows which versions to use for every dependency of every package in that module.

If you don't particularly care about downstream consumers having a package that is provided by the workspace, you can use go mod tidy -e to ignore the error from the missing package.

Otherwise, you either need to publish the workspace dependencies before running go mod tidy (so that they have well-defined upstream versions), or tell the go command what those versions will be (using a replace and require directive in the individual module).

[bcmills]

@matloob, for Go 1.19 I wonder if we should augment the go.work file to be able to declare explicit intended versions for the modules in the workspace. Then go mod tidy could use those versions instead of looking upstream. 🤔

[bozaro]

Currently with reproducing repository (https://github.com/bozaro/go-work-play/tree/go-mod-tidy).

I can run:

cd tools/hello
go run .

And go run works like workspace go modules declared as replace in current tools/go.mod.

But for go mod tidy I need add replace manually.

This behaviour looks like inconsistent: I expects that go run and go mod tidy would use same dependencies.

[liadmord]

What valid use cases are there for go.work if we are still required to fully implement the go.mod in order to work on multi module repo? @bcmills

[AlmogBaku]

This is also happening for go get ./...

I agree with @liadmord, the only benefit I see get from the go.work is a better "understanding" of my IDE Or - maybe I'm doing something wrong here?

[Jictyvoo]

I agree with @bozaro @liadmord and @AlmogBaku

I have a monorepo private project where I have multiple modules, and on each one I need to insert a lot of replace statements for each direct and indirect dependencies. With go workspaces I really want to be possible to run go mod download or go mod tidy without the needed to write a replace statement in every go.mod file in my workspace.

[springeye]

Many of my colleagues have this need

[jincheng9]

a big problem for go 1.18 workspace mode

[maxsxu]

Encountering the same issue with go1.18.2.

Furthermore, some other go mod commands also failed. See sample repo: https://github.com/maxsxu/go-workspace

• go mod vendor

➜  server   go mod vendor 
atmax.io/go-workspace/server imports
        atmax.io/go-workspace/commons/pkg: no required module provides package atmax.io/go-workspace/commons/pkg; to add it:
        go get atmax.io/go-workspace/commons/pkg

• go mod verify

➜  server   go mod verify 
atmax.io/go-workspace/server : missing ziphash: open hash: no such file or directory

Expected behaviour: The go mod commands should not try to find modules which have been declared in go.work. which I think will make go workspace more meaningful.

[Gbps]

My expected functionality is that a go mod tidy on a go.work enabled workspace should feel like a bunch of phantom replace statements inside the go.mod without editing the file. If the module is at github.com/Gbps/mymodule and I tell my go.work that the module can be found in /path/to/mymodule, then go mod tidy should pull the version from my local repo instead of going to github at all.

If I don't commit my go.work, then anyone who uses my go.mod afterwards will have to go to GitHub, which is correct! Now I can edit my local copy and test changes, but when I'm ready to push, I can tidy the correct version into the go.mod and push both the mymodule local dev repo and my current project. This makes sense!

Once I must make an edit to go.mod to include a local path string for any feature of the go mod command line to work, all the purpose of go.work is lost on me. I should never have to taint go.mod with local paths, go mod tidy can find what commit I'm working with locally through go.work and update it accordingly. It should be no different than if I pushed a commit to github and ran go mod tidy to pull the newer version into my go.mod.

In other words, if I add this line to go.mod:

replace github.com/Gbps/mymodule => /path/to/mymodule/

Then go mod tidy works as expected. If I have use /path/to/mymodule/ in my go.work, it should accomplish the same thing. The benefit being I don't have to taint go.mod with my local paths, as that is going to need to be committed to remote whereas go.work isn't.

[opengs]

This is a very big issue for me in the case of creating Docker containers :( I cant use go mod tidy and copy the dependency graph to the Dockerfile. So every time I build a container I have to download downstream dependencies instead of doing go mod download.

Golang is considered the language for microservices and In the current state of the go work it's very hard to work with them in monorepo. Creating multiple-container environment is incredibly hard :(. Basically, every time I hit docker-build it has to download downstream dependencies. Currently, I have two ways to fix this:

1. Push packages to the repo. But if I have ~30 microservices and most of them are 1 file program - it's a very poor experience.
2. replace and require create too many relative dependencies and it's very hard to maintain.

I expect 4 things from the golang workspaces:

1. go mod to know about other modules and don't try to download the local package from the internet. Maybe traverse up in the folder hierarchy and check if go.work exists?
2. you can create a clear dependency graph so it can be used for caching in Docker containers
3. functions like go work tidy and go work download, because now I have to enter every package and type go mod commands. I just want to tidy and download everything from one place, and because we already have a path to the modules in go.work file - it should be trivial.
4. functionality to reduce the number of similar dependencies (download one specific version and force to use it in every module in workspace). This was done by for example turborepo, lerna, and other monorepo management tools.

So now, the only thing why I'm using go work is to have a better codding experience in VsCode. Most of the negative comments that I hear from colleagues about golang are about poor modules/workspaces management. I think, go work is a very important feature to have and it's a very important direction to go.

[HeCorr]

I've faced this issue yesterday but my case is a bit different.. My replace statements are in the go.work file, which seemed to solve go run errors, but when attempting go mod tidy on any of the modules, it is indeed ignored.

The solution is simply to move the replace statements to the go.mod files as explained by Gbps, but keeping track of what package uses what internal package is a bit of a hassle. Is this considered a bug also?

[bcmills]

go mod tidy is intended to update the module's self-contained dependencies. It doesn't use the workspace because in general one may work on multiple independently-maintained modules in the same workspace, and if you're preparing an upstream commit you definitely don't want that commit to rely on unpublished local modifications.

go work sync is intended to update the modules within a workspace to reflect the dependencies selected in that workspace. That is probably what you want if you are working on a set of modules that are all maintained as a single unit.

Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered go work sync for your use-case? If so, what does go work sync do (or not do) that makes it unsuitable for your use-case?

[AlmogBaku]

@bcmills I'm not sure what the intended workflow is, but I think the feedback here is that the DX is really not intuitive.

[burdiyan]

Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered go work sync for your use-case? If so, what does go work sync do (or not do) that makes it unsuitable for your use-case?

I wish there were go work tidy or something similar that would do the same as go mod tidy for each module in the workspace. Or maybe go work sync could have a flag for that, or even do it by default. This way one could easily manage a single repository with many modules, keeping everything in sync and tidy :) without having to switch to every single module directory to keep it tidy.

[lopezator]

There's an additional issue (I think) I reproduced by cloning this repo:

https://github.com/xmlking/go-workspace

If you try to execute:

go mod verify

It will complain with:

github.com/xmlking/go-workspace/cmd/app1 : missing ziphash: open hash: no such file or directory
github.com/xmlking/go-workspace/cmd/app2 : missing ziphash: open hash: no such file or directory
github.com/xmlking/go-workspace/cmd/app3 : missing ziphash: open hash: no such file or directory
github.com/xmlking/go-workspace/lib : missing ziphash: open hash: no such file or directory

There's something we can do to fix this?

[bcmills]

@lopezator, that looks like a separate issue — this one is specifically about go mod tidy.

Could you file it separately (https://go.dev/issue/new)? Thanks.

[mlaventure]

go work sync does not work for me in that scenario either unfortunately. I have a go.work for a monorepo containing yet to be published packages (the domain also does not exist yet). e.g.:

use (
    ./foo
    ./bar
    ./fubar
)

replace not-yet-created.com/foo/foo v0.0.0-00010101000000-000000000000 => ./foo

However that replace directive seems to be ignored by go work sync

$ go work sync
go: not-yet-created.com/foo/[email protected]: unrecognized import path "not-yet-created.com/foo/foo": https fetch: Get "https://not-yet-created/foo/foo?go-get=1": dial tcp: lookup not-yet-created.com on 127.0.0.53:53: server misbehaving

[skelouse]

I've found that the problem with go mod tidy only happens when a sub-module does not exist in a package.

I've created a simple example to demonstrate the issue: https://github.com/skelouse/go-work-test

• Branch master, go run . and go mod tidy work as expected

• Branch success, go run . and go mod tidy both work as expected.

• Branch failing, go run . works, but go mod tidy fails with:

github.com/skelouse/go-work-test/main imports
        github.com/skelouse/go-work-test/foo/newbar: module github.com/skelouse/go-work-test/foo@latest found (v0.0.0-20220809210529-8026891d438c), but does not contain package github.com/skelouse/go-work-test/foo/newbar

I've also created a CLI which uses go.work in combination with vscode workspaces. Currently have to revert back to hard replaces for certain changes.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[nathanblair]

The documentation for workspaces

go.work can be used instead of adding replace directives to work across multiple modules.

seems misleading here if not all of the go commands work with using the go.work as an override for go.mod.

I fully understand and for the most part agree that go mod tidy may not be the solution for addressing tidy-operations in a go workspace tree; a go mod tidy may modify a go.sum with unpublished references that when synced to a remote/upstream may include references that cannot be reached.

This could be a case of cleaning up/amending the documentation with further clarifications on what use-cases exactly a go workspace can solve.

[tliron]

I would like to very humbly suggest that we need a go.work.sum (or similar) file separate from go.sum, as we are dealing with two separate graphs of modules.

[nathanblair]

@tliron Coincidentally something (not sure what tool exactly - it was probably done under the hood from the VS Code Go extension) generated a go.work.sum file in some of my go workspaces.

[nine9ths]

I had unexpected behavior from go get related to go work. I'm concurrently developing a library in a directory and an application in a sibling directory. I did an initial commit on the library on main with just a go.mod file, pushed it to a remote (a private bitbucket instance), then started a develop branch. In the application I created a go.work that looked like:

go 1.19

use (
	.
	../library
)

I am also using vendoring in the application as part of a Docker build process. Everything was working pretty smoothly though, whenever I made a change to the library, I'd commit it, then in the application run:

go get the-module-name/library@hash-of-latest-commit
go mod tidy
go mod vendor

then everything would be up to date. This would work regardless of whether I'd pushed the latest library commit to the remote.

At some point during development, I made a first pull request from the library's develop branch to main on the remote. Once I had done this, if I then tried to go get the library with a local only commit it errors with an invalid commit hash. I now must push the library commits to the remote in order to update them in the application's go.mod or vendor directory. This makes the go.work file irrelevant.

[bcmills]

@tliron, there is already a go.work.sum file that is populated if needed. (In many cases it is not needed — often the individual go.sum files in the workspace are sufficient to cover all of the relevant modules.) But that doesn't really seem relevant to this particular issue.

[bcmills]

@nine9ths, the behavior of go get for an unpublished commit is off-topic for this issue. (That's more closely related to #28835.)

[tliron]

@bcmills I did not know this and do not see this file in our environment. Could you point me to the documentation where go.work.sum is discussed? When is it populated? I assume that if a go.work.sum exists, then it would be used instead of go.sum for all operations?

Specifically to our topic, when I run go mod tidy does it update both go.sum and go.work.sum at the same time? Would there be a way to update only go.work.sum? That last ability would solve all our problems, which is why I raised the topic. During work, go mod tidy would fail and we know it would fail because ... we are in the midst of working on it. :)

We're having a lot of trouble with this go.work feature, like others here.

[bcmills]

@tliron, documentation for go.work.sum is #51941. Again, please keep discussion on the issue tracker specific to the issue.

[keilin-anz]

Thought I found a workaround, but it's broken in its own unique ways see here

Turns out go mod tidy will happily skip attempting to download modules which don't look like URLs - which seemed to have some potential for creative misuse...

So you can use local module names like my-repo/my-module instead of github.com/my-repo/my-module just fine...

go mod tidy behaves itself and doesn't try to download remote versions of local workspace modules go run / go build work properly go work sync works properly...

until someone else wants to import those modules (or any modules in your monorepo which depend upon them)

Seems like all the building blocks are there to support what's discussed in this issue, but there's logic in various tooling which prevents it. (eg. if go get didn't care if your internal module names matched how they're presented to the world via the go.work file it would be fine.)

Perhaps worth considering either:

1. Change the default behaviour - potentially incurring the wraith of people who have monorepo's in which the contained modules depend on different versions of their own internal modules (is that a thing?)
2. Allow overriding the default behaviour from go.work so that the go.mod files remain intact and behave as expected

[spekary]

@bcmills go work sync only works if you actually have a go.work file. Otherwise it errors.

I have a scripted install process that involves calling go mod tidy followed by a number of go install calls, all part of a continuous integration process.

I was using go.work to be able to test the code in a working branch. However, go mod tidy attempts to grab the code from a release on a main branch. Normally, I want that, but when testing new code in a working branch, I don't.

Trying to use go work sync would mean I would have to change the install scripts to do a go work sync if a go.work file was available, and go mod tidy if not. Not too difficult, but a pain.

However, running go work sync on a new install gives me errors like this:

require github.com/goradd/goradd: version "latest" invalid: must be of the form v1.2.3

So go work sync will not do what is expected, which is to get the latest versions of the packages referred to, except if there is a use directive, in which case it should just use whatever is pointed to. This means I am stuck. I can't call go mod tidy because it will error on a missing package since it will not use the go.work file to find it and so produces an error. And I can't call go work sync because it will not load the latest packages when the go.mod file is asking for it, and so produces an error. Perhaps I am missing an argument that will enable updating of the go.mod file? go help work sync and go work help did not reveal anything helpful in this regard.