proposal: cmd/go: track tool dependencies in go.mod

[mtibben]

Background

The current best-practice to track tool dependencies for a module is to add a tools.go file to your module that includes import statements for the tools of interest. This has been extensively discussed in #25922 and is the recommended approach in the Modules FAQ

This approach works, but managing the tool dependencies still feels like a missing piece in the go mod toolchain. For example, the instructions for getting a user set up with a new project using gqlgen (a codegen tool) looks like this

# Initialise a new go module
mkdir example
cd example
go mod init example

# Add gqlgen as a tool
printf '// +build tools\npackage tools\nimport _ "github.com/99designs/gqlgen"' | gofmt > tools.go
go mod tidy

# Initialise gqlgen config and generate models
go run github.com/99designs/gqlgen init

The printf line above really stands out as an arbitrary command to "add a tool" and reflects a poor developer experience when managing tools. For example, an immediate problem is that the printf line will only work on unix systems and not windows. And what happens if tools.go already exists?

So while we have some excellent tools for managing dependencies within the go.mod file using go get and go mod edit, there is no such equivalent for managing tools in the tools.go file.

Proposed Solution

The go.mod file uses the // indirect comment to track some dependencies. An // indirect comment indicates that no package from the required module is directly imported by any package in the main module (source).

I propose that this same mechanism be used to add tool dependencies, using a // tool comment.

Users could add a tool with something like

go get -tool github.com/99designs/[email protected]

or

go mod edit -require=github.com/99designs/gqlgen -tool

A go.mod would then look something like

module example

go 1.17

require (
	github.com/99designs/gqlgen v0.14.0 // tool
)

And would allow users to subsequently run the tool with go run github.com/99designs/gqlgen

This would mean a separate tools.go file would no longer be required as the tool dependency is tracked in the go.mod file.

Go modules would be "tool" aware. For example:

• go mod tidy would not remove the // tool dependency, even though it is not referenced directly in the module
• Perhaps if a module with a // tool dependency is imported by another module, Go modules understands that the // tool dependency is not required as an indirect dependency. Currently when using tools.go, go modules does not have that context and the tool is treated like any other indirect dependency
• go get -tool [packages] would only add a dependency with a main package

[fsouza]

I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

[ianlancetaylor]

CC @bcmills @jayconrod

[mtibben]

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

[mvdan]

Personally, I think https://github.com/golang/go/issues/42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

[mvdan]

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

[seankhliao]

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed. I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

[mtibben]

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

1. You mean go run hack.me/[email protected] will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect

2. Should gqlgen instructions always be to specify version with go run github.com/99designs/[email protected]? That seems verbose

3. Repetition across multiple files, keeping version in sync, yep your comment above nails it

[mtibben]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

[mvdan]

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

[mvdan]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

[mtibben]

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

[mtibben]

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

[lwc]

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

[fsouza]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

[bcmills]

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

[mvdan]

@bcmills would such tool requirements be part of the same MVS module graph?

[bcmills]

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

• go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
• go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
• go list direct (#40364) would report the named packages as direct imports.

[carldunham]

Or go list tools

[jayconrod]

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

[mtibben]

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

[mtibben]

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

[jayconrod]

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

[mtibben]

Yep that makes a lot of sense @jayconrod

[rsc]

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

[mtibben]

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

[jayconrod]

Sure, I'll paste my draft proposal below. Unfortunately I won't be able to work on the implementation for this, but this is what I was thinking in terms of design.

---

tool directive

I propose adding a tool directive to the go.mod file format. Each tool directive names a package.

tool golang.org/x/tools/cmd/stringer

go mod tidy and go mod vendor would act as if each tool package is imported by a package in the main module. Tool packages would be matched by the all metapackage.

Modules providing tool packages must still be required with require directives. Requirements for tools would not be treated differently from other requirements. This means that if a command and a library are needed from the same module, they must be at the same version (related: #33926). Requirements on modules providing tools would also affect version selection in dependent modules if lazy loading is not enabled.

The tool directive itself would not affect version selection. go mod tidy, go mod vendor, and other commands would ignore tool directives outside the main module.

go get -tool

tool directives could be added or removed with go get, using the -tool flag. For example:

go get -tool golang.org/x/tools/cmd/[email protected]

The command above would add a tool directive to go.mod if one is not already present. It would also add or update the requirement on golang.org/x/tools and any other modules implied by that update.

require golang.org/x/tools v0.1.0
tool golang.org/x/tools/cmd/stringer

A tool directive could be removed using the @none version suffix.

go get -tool golang.org/x/tools/cmd/stringer@none

-tool could be used with -u and -t.

tools metapackage

To simplify installation, go install and other commands would support a new metapackage, tools, which would match packages declared with tool dependencies.

# Install all tools in GOBIN
go install tools

# Install all tools in the bin/ directory
go build -o bin/ tools

# Update all tools to their latest versions.
go get tools

It would not be an error for a tool directive to refer to a package in the main module, so the tools metapackage could match a mix of local and external commands.

[mtibben]

Modules providing tool packages must still be required with require directives. Requirements for tools would not be treated differently from other requirements. This means that if a command and a library are needed from the same module, they must be at the same version (related: #33926). Requirements on modules providing tools would also affect version selection in dependent modules if lazy loading is not enabled.

Ah yes this is similar to the // tools approach which wouldn't allow a different version between the tool binary and the library to be specified.

Specifying the actual location of the binary isn't something the // tools approach solves - but do we need to? In the // tools approach, version is specified once as part of a normal require directive, but the the location binary to be run is left up to go run. e.g. the go.mod would look like

require golang.org/x/tools v0.1.0 // tool

and you'd call go run which would use the version defined by go.mod

go run golang.org/x/tools/cmd/stringer

@jayconrod Is the reason for specifying the exact tool location as you've described (e.g.tool golang.org/x/tools/cmd/stringer) just to allow go install tools? I feel that "installing" tools might just cause the same kind of version issues between projects if installing to a common location, and go run may be the superior approach.

[jayconrod]

@jayconrod Is the reason for specifying the exact tool location as you've described (e.g.tool golang.org/x/tools/cmd/stringer) just to allow go install tools? I feel that "installing" tools might just cause the same kind of version issues between projects if installing to a common location, and go run may be the superior approach.

I was suggesting tools would be a metapackage (like cmd or std), so you could use it with go install tools (hopefully setting GOBIN first), or go build -o bin/ tools or go list tools; it would work with any subcommand that accepts package arguments.

I think having a separate tool directive in go.mod is helpful for understanding why a requirement is needed. For example, suppose you stop using golang.org/x/tools/cmd/stringer in favor of a more advanced tool. At some point in the future, you might see:

require golang.org/x/tools v0.1.0 // tool

and wonder why it's there. It's not clear that it's safe to remove. But with:

tool golang.org/x/tools/cmd/stringer

you'd know that it's not used anymore, so it's safe to delete that line. The next go mod tidy would remove the corresponding requirements (assuming no other packages are needed).

[bwplotka]

2c from my side: Go tools are used not only by Go modules. This is why a separate go.mod for tools or something like what https://github.com/bwplotka/bingo automated for you might be preferred. (:

[jayconrod]

@bwplotka Separate go.mod files are already supported via the -modfile flag since Go 1.14.