rules_go
rules_go copied to clipboard
bazelbuild
Expose generated Go files to editors and IDEs
When a library is created with go_proto_library or some other mechanism that generates Go code (.pb.go files in this case), the generated sources files are not exposed to editors and IDEs. Consequently, code completion and refactoring tools don't work, lowering developer productivity.
The Go team is developing a workspace abstraction mechanism that will allow editors and Bazel (and other build systems) to communicate with each other without the need for direct integration. This is the long-term solution to this problem.
This issue will track progress on the workspace abstraction and will track editor awareness of code generated with Bazel.
This issue is in response to Bazel build, protobuf and code completion.
TODO: update answer when this issue is resolved.
So, I have proposal how to close this issue.
Suppose you generated code: bazel-genfiles/somedir/somefile.go bazel-genfiles/somedir/anotherfile.go
Let's add symlink src/somedir => ../bazel-genfiles/somedir
Use in your files
import "somedir"
As result you will get workable code-completition AND workable build.
@excavador We're still developing a more complete solution to this (the workspace abstraction I mentioned above). Ideally, tools won't need to be aware of the project layout in use. I don't have any ETA for that yet, but we are making progress.
As you showed, it's possible to work around this with symlinks. You can also copy or symlink generated files into your source directory. Bazel will prefer a generated file if a source file of the same name is present. I'd recommend against checking generated files into VCS though.
Thank you for notify! Will wait. You can consider me as closed beta-tester :)
@raliste it works for me in IDEA Ultimate + Bazel plugin with following limitation: autocomplete works only if all files inside golang package are auto-generated. If mix together auto-generated and manually written file, autogenerated file would not be inspected
The workspace abstraction mentioned above is being actively worked on. We've gone through a lot of internal prototypes, and we're in the process of reviewing, testing, and merging one that we're pretty happy with. It's a huge body of code though, and it will take time. Following that, a large number of Go tools will need to be updated to use the workspace abstraction so they can function in "go build", vgo, Bazel, and other kinds of workspaces.
I know it seems like progress is slow, but it's an interface that many, many tools will interact with, and we want to be sure we get it right. It's a huge amount of work, but it is our highest priority task.
@jayconrod Are there any public discussions about that? Design docs perhaps?
Not yet, but soon. We're still working on simplifying and narrowing the interface.
Can you briefly explain how this workspaces feature is going to work? Are there any upcoming changes in the 1.11 related to this?
The first public piece of this is golang.org/x/tools/go/packages. At the moment, that is based on go list and is intended to be used by tools that support go build and vgo. We're also working on an workspace abstraction layer that go/packages may eventually be built on. That workspace abstraction would be extensible and could include Bazel. Lot of work still happening there.
@akshayjshah We've made some progress. The golang.org/x/tools/go/packages API is nearing completion. Our team has migrated a few tools to use it already, but the effort is ongoing.
If you set the GOPACKAGESDRIVER environment variable, or if you have a binary named gopackagesdriver in PATH, go/packages will shell out to that binary for queries. I have an implementation of that binary about 70% done. It works in most situations, but it needs polish, documentation, and tests.
So ideally, once that change is in, you'd be able to build and install the driver into your PATH, and tools that use go/packages will just work in Bazel workspaces (and they'll fall back to the default behavior if there's no WORKSPACE file).
Thanks for the update! Would you be willing to push your driver to a branch so I can take a look? I'm not in a huge rush for you to officially release it, but I'd love to use your rough draft to better understand how these pieces should fit together.
@akshayjshah Sure, it's at https://github.com/jayconrod/rules_go/tree/dev-gopackagesdriver if you'd like to take a look. As I said, it's very rough right now.
@pierreis Sorry, no updates since my last comment. I'm still planning to work on this this quarter. There are a lot of things going on, but this is one of the higher priority projects.
Generated .pb.go files are now available in the go_generated_srcs output group since #1827.
Leaving this issue open though since it's not documented yet.
@jayconrod is package driver still planned as a feature or are we going to rely on the aspect-based approach mentioned above?
A driver is still planned, but I don't have any update to share yet. Implementation-wise, the data will be produced by the builder and accessed through an output group (but the driver will hide those details).
@jayconrod rebased your branch and updated it with the latest changes from the go/packages API to play around a bit with the driver concept. I managed to get it to a point where I can use the bazel driver to power some of the module enabled tools like godef-mod or gopls. You can find it at https://github.com/robbertvanginkel/rules_go/tree/dev-gopackagesdriver.
Some of the code is a bit gnarly, needs some tests and there's still a few TODO's in there, but I'd love to help out and get the driver added to rules_go. Since the branch is mostly based on your changes, do you have a suggestion on how we could collaborate on this?
@robbertvanginkel I'm probably not going to keep most of the code on that branch. I don't think gathering information through separate actions via an aspect will be fast enough, and it adds a lot of complication. Instead, I'm thinking of having the compile action emit JSON metadata, then the driver can just "build" an output group and collect the results.
But a fair bit of refactoring is needed on the builder side of things first. I have a half-written PR to pull everything into a single binary and have the toolchain own that.
In short, I'm not ready to collaborate on this yet.
Ok, thanks for the update! Please me know if you reach a point where someone can help out.
@jayconrod I’m newbie in Skylark (and bazel concepts) yet. Can you tell how to copy (or link) files described in OutputGroup go_generated_src to some folder inside the workspace? In other words, can I now generate .pb.go a la protoc + protoc-gen-go using rules_go?
@asv This is something I wish were simpler in Bazel. You can build with the command line argument --output_groups=go_generated_src (or any comma separated list of group names). To get the paths to the files in a machine-readable format, you can use --build_event_json_file or --build_event_proto_file, depending on whether you want to parse JSON or proto. build_event_stream.proto describes the format. You'll be looking for an event with a TargetComplete payload.
@jayconrod I'm got it!
Small fix to your post: --output_groups=go_generated_srcs. Name of group go_generated_srcs, not go_generated_src. It's a pity that bazel does not complain on error in name.
What's the state of this ticket? It looks like the necessary functions were added to Bazel. It also looks like there's a go package to support it. What's missing?