cmd/go: add global ignore mechanism for Go tooling ecosystem

[burdiyan]

UPDATE: The summary of the accepted proposal at: https://github.com/golang/go/issues/42965#issuecomment-1974089649.

---

Problem

For non-trivial (often multi-language) projects it's often desirable to make all the Go tools (including gopls) ignore certain directories.

Some of the examples could be the huge amount of files within node_modules, or bazel-* directories generated by Bazel. This causes many operations with ./... wildcards taking longer than desired. Also gopls often eats up a lot of CPU in VS Code depending on what you are doing.

Prior Art

This is something that has been discussed in several issues before, but seems like people couldn't get agree on a solution.

• https://github.com/golang/go/issues/30058
• https://github.com/golang/go/issues/35914
• https://github.com/golang/go/issues/42473

Some tools started to have their own solutions which causes fragmentation and is cumbersome.

For example goimports have its own machinery for this - .goimportsignore file in this case. But it's not working with Go Modules.

Other tools have a hard-coded list of directories to ignore, like .git and so on.

It seems like having a global solution that all the Go ecosystem could understand would make sense to solve this kind of problem.

Recently a workaround for this was to place a dummy go.mod file in the directories you wanted to ignore. But this is not easily portable between users of the project, because often these directories can be re-created on the user's machine and aren't even checked-in. Asking people to sprinkle some go.mod files all around every time is cumbersome.

@robpike was against of creating more dot files (https://github.com/golang/go/issues/30058#issuecomment-475003231).

Proposed Solution

Here're some of the options that this could be implemented with.

1. ~~Use go.mod file for specifying directories to ignore.~~ (Rejected because go.mod is not a catch-all config file like package.json in NodeJS).
2. ~~Use a separate .goignore file.~~ (This would go against Rob's desire to avoid new dot files, and although being in the spirit with other tools: .dockerignore, .gitignore, .bazelignore, etc. is concerning. The concerns are discussed in this thread).
3. Use the go.work file that's coming in the next Go 1.18 release.
4. Have a separate go.ignore file that would specify directories to ignore.

/cc @tj @stamblerre