Cannot compile //:gazelle when using version 0.25.0

[graywolf-at-work]

What version of gazelle are you using?

0.25.0

What version of rules_go are you using?

0.31.0

What version of Bazel are you using?

5.0.0

Does this issue reproduce with the latest releases of all the above?

Yes.

What operating system and processor architecture are you using?

Linux, amd64

What did you do?

+$ cat BUILD
load("@bazel_gazelle//:def.bzl", "gazelle")
gazelle(name = "gazelle")
+$ cat WORKSPACE
workspace(name = "foobar")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")

maybe(
    http_archive,
    name = "io_bazel_rules_go",
    urls = [
        "https://github.com/bazelbuild/rules_go/releases/download/v0.31.0/rules_go-v0.31.0.zip",
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.31.0/rules_go-v0.31.0.zip",
    ],
    sha256 = "f2dcd210c7095febe54b804bb1cd3a58fe8435a909db2ec04e31542631cf715c",
)
load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")

maybe(
    http_archive,
    name = "bazel_gazelle",
    urls = [
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.25.0/bazel-gazelle-v0.25.0.tar.gz",
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.25.0/bazel-gazelle-v0.25.0.tar.gz",
    ],
    sha256 = "5982e5463f171da99e3bdaeff8c0f48283a7a5f396ec5282910b9e8a49c0dd7e",
)
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")

############################################################
# Define your own dependencies here using go_repository.
# Else, dependencies declared by rules_go/gazelle will be used.
# The first declaration of an external repository "wins".
############################################################

go_rules_dependencies()
go_register_toolchains(version = "1.18")
gazelle_dependencies()
+$ bazel build --verbose_failures //:gazelle
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:gazelle (81 packages loaded, 8685 targets configured).
INFO: Found 1 target...
ERROR: /home/wolf/.cache/bazel/_bazel_wolf/6dd1b1f68dde97d4c1e29873f38ce7a7/external/org_golang_x_mod/module/BUILD.bazel:3:11: GoCompilePkg external/org_golang_x_mod/module/module.a [for host] failed: (Exit 1): builder failed: error executing command
  (cd /home/wolf/.cache/bazel/_bazel_wolf/6dd1b1f68dde97d4c1e29873f38ce7a7/sandbox/linux-sandbox/17/execroot/foobar && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src external/org_golang_x_mod/module/module.go -src external/org_golang_x_mod/module/pseudo.go -arc 'golang.org/x/mod/internal/lazyregexp=golang.org/x/mod/internal/lazyregexp=bazel-out/host/bin/external/org_golang_x_mod/internal/lazyregexp/lazyregexp.x' -arc 'golang.org/x/mod/semver=golang.org/x/mod/semver=bazel-out/host/bin/external/org_golang_x_mod/semver/semver.x' -importpath golang.org/x/mod/module -p golang.org/x/mod/module -package_list bazel-out/host/bin/external/go_sdk/packages.txt -o bazel-out/host/bin/external/org_golang_x_mod/module/module.a -x bazel-out/host/bin/external/org_golang_x_mod/module/module.x -gcflags '' -asmflags '')
# Configuration: ad7becf1f627c98f4c44ac49f5991ab41aa98662628ab4c26b029d7fb1035392
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
compilepkg: missing strict dependencies:
        /home/wolf/.cache/bazel/_bazel_wolf/6dd1b1f68dde97d4c1e29873f38ce7a7/sandbox/linux-sandbox/17/execroot/foobar/external/org_golang_x_mod/module/module.go: import of "golang.org/x/xerrors"
No dependencies were provided.
Check that imports in Go sources match importpath attributes in deps.
Target //:gazelle failed to build
INFO: Elapsed time: 68.110s, Critical Path: 1.56s
INFO: 32 processes: 18 internal, 14 linux-sandbox.
FAILED: Build did NOT complete successfully

What did you expect to see?

The build to succeed.

What did you see instead?

The build failed.

[lukedirtwalker]

Seems manually adding the xerrors dependency helps:

go_repository(
    name = "org_golang_x_xerrors",
    importpath = "golang.org/x/xerrors",
    sum = "h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=",
    version = "v0.0.0-20200804184101-5ec99f83aff1",
)

This seems to be explicitly overriding the version that would otherwise come from rules_go.

[graywolf-at-work]

The suggested override does help, so thank you for that.

gazelle \ rules_go | 0.30.0 | 0.31.0 |
-------------------+--------+--------+
      0.24.0       |   OK   |   OK   |
      0.25.0       |  FAIL  |  FAIL  |
-------------------+--------+--------+

However, I still struggle to see what am I doing so special that it's not working for me. I assume the 0.25.0 was tested and I assume other people use it, so any ideas why it's not working just for me?

[lukedirtwalker]

Actually for us it also doesn't really work. The workaround only partially helped but then with clean cache builds fail.

[nickgooding]

Edit: After some more messing around, I found that placing this in my WORKSPACE was sufficient to fix the problem

# gazelle:repository go_repository name=org_golang_x_xerrors importpath=golang.org/x/xerrors

---

This seems to be related to the new static dependency resolution mode (9c47513). I had success applying this patch to Gazelle:

diff -urN a/deps.bzl b/deps.bzl
--- a/deps.bzl
+++ b/deps.bzl
@@ -317,6 +317,7 @@ def gazelle_dependencies(
     _maybe(
         go_repository,
         name = "org_golang_x_mod",
+        build_external = "external",
         importpath = "golang.org/x/mod",
         sum = "h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=",
         version = "v0.6.0-dev.0.20220106191415-9b9b3d81d5e3",

Without the patch (i.e. using static dependency resolution mode), the generated @org_golang_x_mod//module:module target is missing the @org_golang_x_xerrors//:go_default_library dependency which causes the "missing strict dependencies" error.

[kczulko]

Thank you @nickgooding

Your solution works. I just took this diff and applied it with patch_args = ["-p1"],. Now my project works!

[chrismgrayftsinc]

To expand on @nickgooding's comment, if you use gazelle from within bazel, then putting the following in the WORKSPACE file worked for me:

############################################################
# Define your own dependencies here using go_repository.
# Else, dependencies declared by rules_go/gazelle will be used.
# The first declaration of an external repository "wins".
############################################################

go_repository(
    name = "org_golang_x_mod",
    importpath = "golang.org/x/mod",
    sum = "h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=",
    version = "v0.6.0-dev.0.20220106191415-9b9b3d81d5e3",
    build_external = "external",
)

go_rules_dependencies()

[yohcop]

Including build_external = "external" works for me as well. But... I'm really looking forward to the accelerated/cached generation of packages :) . It sometimes takes more than 15 minutes to run gazelle on cloud.google.com/go.

It's admittedly a bit of a weird package, as it's huge. Even though I only use 3 APIs in it out of the ~100. Maybe there's a trick to ignore some sub directories? (I could create a patch deleting most directories maybe... Edit: I tried adding all the extra directories with something like build_extra_args=["-exclude=bigtable", "-exclude=automl", ...], but it doesn't seem to help)

[sluongng]

In https://github.com/sluongng/nogo-analyzer I had a really hard time compiling the repo after adding golangci-lint, another huge repo, as a direct dependency.

Adding build_external = "external" to each failing to compile go_repository target eventually become unsustainable thus I had to just set everything to external by default.

Please feel free to use my repo to test/reproduce if needed. A simple sed to remove all build_external = "external", lines should immediately highlight the problem.

[tjoneslo]

After doing some digging I found #1210 is the source of this problem. All the commits prior to this one work correctly without any changes to our code, but with this change see the missing strict dependencies errors on different packages.

[sluongng]

FYI I created https://github.com/bazelbuild/bazel-gazelle/pull/1264 to switch back to external dep resolution mode by default. Imo it is a more sensible default for users, at least until static dep resolution mode could resolve the issues reported in this thread.

[matzf]

For what it's worth, this issue is not reproducible with the latest rules_go v0.33.0 release. Judging just from the release notes, this is likely due to the updated org_golang_x_xerrors, just like @lukedirtwalker had proposed above as a workaround.

Nope, sorry. The problem persists -- I thought I was doing the same thing as described in the issue, but wasn't. When that worked, I jumped to the most ready explanation, but this was not correct. Sorry!

[sluongng]

~For what it's worth, this issue is not reproducible with the latest rules_go v0.33.0 release. Judging just from the release notes, this is likely due to the updated org_golang_x_xerrors, just like @lukedirtwalker had proposed above as a workaround.~

Nope, sorry. The problem persists -- I thought I was doing the same thing as described in the issue, but wasn't. When that worked, I jumped to the most ready explanation, but this was not correct. Sorry!

I tested it in my repo yesterday and I was able to build everything without build_external, so I think you are right that new rules_go version might have fixed this. https://github.com/sluongng/nogo-analyzer/commit/21610b27d2d0219610e43b0d8bbaa263786f438d

However, i think that org_golang_x_xerrors and org_golang_x_mod are very common dependencies in all projects and thus can be easily overriden inside several WORKSPACE macro calls. User can troubleshoot this by doing this query.

bazel query 'set(//external:org_golang_x_mod //external:org_golang_x_xerrors)' --output=build | grep -v '#'

go_repository(
  name = "org_golang_x_mod",
  importpath = "golang.org/x/mod",
  version = "v0.6.0-dev.0.20220106191415-9b9b3d81d5e3",
  sum = "h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=",
  build_external = "external",
)

go_repository(
  name = "org_golang_x_xerrors",
  generator_name = "org_golang_x_xerrors",
  generator_function = "ugazelle_deps",
  importpath = "golang.org/x/xerrors",
  version = "v0.0.0-20200804184101-5ec99f83aff1",
  sum = "h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=",
  build_external = "external",
)

[whs-dot-hk]

@chrismgrayftsinc I can confirm your temp fix works. Below is my WORKSPACE file

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "685052b498b6ddfe562ca7a97736741d87916fe536623afb7da2824c0211c369",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.33.0/rules_go-v0.33.0.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.33.0/rules_go-v0.33.0.zip",
    ],
)

http_archive(
    name = "bazel_gazelle",
    sha256 = "5982e5463f171da99e3bdaeff8c0f48283a7a5f396ec5282910b9e8a49c0dd7e",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.25.0/bazel-gazelle-v0.25.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.25.0/bazel-gazelle-v0.25.0.tar.gz",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository")

go_repository(
    name = "org_golang_x_mod",
    build_external = "external",
    importpath = "golang.org/x/mod",
    sum = "h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=",
    version = "v0.6.0-dev.0.20220106191415-9b9b3d81d5e3",
)

go_rules_dependencies()

go_register_toolchains(version = "1.18")

gazelle_dependencies()

[sluongng]

            compilepkg: missing strict dependencies:
                /private/var/tmp/_bazel_sngoc/7dc6b2f9a9dfc0f9597820642aefdeec/sandbox/darwin-sandbox/287/execroot/__main__/external/co_honnef_go_tools/analysis/facts/tokenfile/token.go: import of "golang.org/x/tools/go/analysis"
                
            compilepkg: missing strict dependencies:
                /private/var/tmp/_bazel_sngoc/7dc6b2f9a9dfc0f9597820642aefdeec/sandbox/darwin-sandbox/352/execroot/__main__/external/co_honnef_go_tools/config/config.go: import of "github.com/BurntSushi/toml"
            No dependencies were provided.

            compilepkg: missing strict dependencies:
                /private/var/tmp/_bazel_sngoc/7dc6b2f9a9dfc0f9597820642aefdeec/sandbox/darwin-sandbox/362/execroot/__main__/external/co_honnef_go_tools/go/ast/astutil/util.go: import of "golang.org/x/exp/typeparams"
            No dependencies were provided.

I still get some issues with dependency resolve of org_golang_x_tools.

Adding this to WORKSPACE file manually tell gazelle how to resolve the dependency which works as a lightweighted workaround in some cases.

# gazelle:repository go_repository name=org_golang_x_tools importpath=golang.org/x/tools
# gazelle:repository go_repository name=com_github_burntsushi_toml importpath=github.com/BurntSushi/toml
# gazelle:repository go_repository name=org_golang_x_exp_typeparams importpath=golang.org/x/exp/typeparams

These deps are being loaded into WORKSPACE from a macro in a separate file, so I wonder if that affected the static resolution somehow?

[tommyknows]

Regarding @sluongng's last hint

These deps are being loaded into WORKSPACE from a macro in a separate file, so I wonder if that affected the static resolution somehow?

Because I also ran into this issue, I removed that external file and let Gazelle write into my WORKSPACE instead. After running update-repos, gazelle removed all my external dependencies from go_repository rules, for whatever reason, but running a normal gazelle update seems to have fixed that as well.

After looking at the dependency resolution docs, I think the behaviour we're seeing is even documented:

In static mode, Gazelle has the same behavior as external mode, except that it will not call out to the network for resolution when no matching import is found within WORKSPACE. Instead, it will skip the unknown import. This is the default mode for go_repository rules.

Now this is a bit nit-picky, but it says is found within WORKSPACE, which with Macros is not the case.

My build also seems to have worked, so I can confirm that this statement seems to be true :-)

Imo gazelle should obviously also look within the external file that is referenced for the macro.

[whs-dot-hk]

The latest version of rules_go and bazel-gazelle works out of the box

Name	Version
rules_go	0.33.0
bazel-gazelle	0.26.0

[Choongkyu]

The latest version of rules_go and bazel-gazelle works out of the box

Name Version rules_go 0.33.0 bazel-gazelle 0.26.0

I still seem to get this error after updating WORKSPACE to use these versions. I still ned to manually add

go_repository(
    name = "org_golang_x_xerrors",
    importpath = "golang.org/x/xerrors",
    sum = "h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=",
    version = "v0.0.0-20200804184101-5ec99f83aff1",
)

to resolve this issue

[garymm]

Using rules_go v0.34.0 and gazelle v0.26.0, I get these errors when my go_repository() calls are in a macro, but not when the same calls are in the WORKSPACE. Should I file a new issue about this?

[appuser999]

rules_go v0.33.0 and gazelle v0.26.0, same errors.