gopls - generated go files - tree artifacts

[lromor]

What version of rules_go are you using?

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
    ],
)

What version of gazelle are you using?

Not using it

What version of Bazel are you using?

6.0.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux, NixOs, amd64

Any other potentially useful information about your toolchain?

Local build, nothing special

What did you do?

I wrote a special rule to generate golang files within bazel. The set of generated files is unknown except at runtime. I am aware of actions.declare_directory as a solution which indeed works. The only problem is that it seems the gopackagedriver doesn't seem to be able to pickup that the returned depset is actually a folder hence (I guess) it doesn't resolve to the right location and fails to index the generated files.

By manually enumerating the files and defining them during action.run_shell everything works fine!

What did you expect to see?

To gopls find the generated package.

What did you see instead?

An error!

[fmeum]

If you provide me with a small Git repo and setup instructions that reproduce the issue, I would work on a fix.

[lromor]

Sure thing! It will take some time though. In the meantime, I have the feeling I got the gist of the problem:

The GoFiles field https://github.com/bazelbuild/rules_go/blob/master/go/tools/gopackagesdriver/aspect.bzl#L53-L56 is missing if the source file is declared with https://bazel.build/rules/lib/actions#declare_directory.

    out_dir = ctx.actions.declare_directory(ctx.attr.name + ".go")
    args = ["--target", out_dir.path, "-package", pkg, "--clean", ctx.file.src.path]
    ctx.actions.run_shell(
        inputs = inputs,
        outputs = [out_dir],
        tools = [
            ctx.file.openapi_tool,
            go_ctx.go,
        ],
        command = """
           trap 'rm -rf {gopath}/_dereferenced' EXIT &&
           mkdir -p {gopath}/_dereferenced/ &&
           mkdir -p {outpath} &&
           cp -rL {gopath}/src {gopath}/_dereferenced/ &&
           source <($PWD/{godir}/go env) &&
           export PATH=$GOROOT/bin:$PWD/{godir}:$PATH &&
           export GOPATH={gopath}/_dereferenced &&
           mkdir -p .gocache &&
           export GOCACHE=$PWD/.gocache &&
           {cmd} {args}
        """.format(
            godir = go_ctx.go.path[:-1 - len(go_ctx.go.basename)],
            gopath = gopath,
            outpath = out_dir.basename,
            cmd = "$(pwd)/" + ctx.file.openapi_tool.path,
            args = " ".join(args),
            mnemonic = "GoOGenSourceGen",
        ),
        progress_message = "Generating ogen go files into '%s'" % out_dir.path,
        env = {
            "GO111MODULE": "off",  # explicitly relying on passed in go_path to not download modules while doing codegen
        },
    )
    return [DefaultInfo(
        files = depset([out_dir]),
    )]

But now the path written in GoFiles and in the json is the path of the directory ending in .go instead of its paths contents. https://github.com/bazelbuild/rules_go/blob/master/go/tools/gopackagesdriver/aspect.bzl#L72

A solution could be to generate the json using an ctx.actions.run_shell with an https://bazel.build/rules/lib/Args#add_all to expand the name of those source paths!

What do you think?

[fmeum]

Thanks for the investigation, that sounds exactly right. It seems like you know pretty well what would need fixing - would you consider working on a PR for this? I can support you when you get stuck.

[lromor]

Ofc! I'd love to contribute!

[lromor]

My current approach would be use go_tool_binary to run some go program to generate the json. This because:

1. I must use run_shell to expand Args.
2. We want to have a portable solution.

I have a question though, what is the semantic difference betwee GoFiles and CompiledGoFiles? I'm just wondering if I should make the resolution work in both cases. What's your view on that?

[fmeum]

A small Go program that generates the JSON sounds reasonable. In that case you should be able to use ctx.actions.run instead of ctx.actions.run_shell.

GoFiles and CompiledGoFiles only differ when code generation (cgo or go coverage instrumentation) is used. GoFiles are probably the more important ones, but if you can handle both, that may still be useful in some cases.

[ChrisVandoo]

@fmeum I've run into a similar issue and I see there is a PR up with a fix (which you've approved) that would resolve this issue. Is there a timeline on when this might be merged?