Determining whether a library is a module or a package for multi-module repositories seems to depend on HEAD

[thompsonja]

What version of gazelle are you using?

v0.23.0

What version of rules_go are you using?

v0.29.0

What version of Bazel are you using?

v4.2.2

Does this issue reproduce with the latest releases of all the above?

Yes, I did try it with gazelle v0.24.0

What operating system and processor architecture are you using?

gLinux, x86_64

What did you do?

I built a project a couple days ago and today, changed nothing, and now builds fail.

What did you expect to see?

No change (builds still succeed).

What did you see instead?

Errors like:

ERROR: /usr/local/google/home/thompsonja/.cache/bazel/_bazel_thompsonja/96b903f5c9e799018aa78e5791da1d42/external/org_golang_x_oauth2/google/BUILD.bazel:3:11: no such package '@com_google_cloud_go_compute//metadata': The repository '@com_google_cloud_go_compute' could not be resolved and referenced by '@org_golang_x_oauth2//google:google'

What I think happened is that the compute/metadata library (https://pkg.go.dev/cloud.google.com/go/compute/metadata) was only made a module today or yesterday.

Even though I'm using v0.81.0 of the cloud.google.com/go library in my bazel dependencies (where compute/metadata is not a module), gazelle seems to be looking at the latest version of cloud.google.com/go in order to create the dependencies. This means that yesterday, a package (golang.org/x/oauth2 in this case) that depended on the compute/metadata library was resolving the dependency to @com_google_cloud_go//compute/metadata:go_default_library, which worked fine, to now being resolved as @com_google_cloud_go_compute//metadata:go_default_library, which doesn't work since com_google_cloud_go_compute was not defined in my WORKSPACE.

The quick solution is just add com_google_cloud_go_compute into my WORKSPACE, but this is also annoying long-term since it implies that this can change regardless of how your own WORKSPACE dependencies are pinned. This also applies to the iam module which was created around the same timeframe.

[minor-fixes]

Seeing this same issue, at exactly the same time in our repo. It seems like a server-side change has broken builds from otherwise-unchanged source.

[minor-fixes]

Additional context: we have two repos (repo A and internal repo B)

Repo A actually works after bazel clean --expunge && bazel query 'deps(//...)' - the above issue is NOT seen. The generated build file @org_golang_x_oauth2//google/BUILD.bazel is:

load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")

go_library(
    name = "google",
    srcs = [
        "appengine.go",
        "appengine_gen2_flex.go",
        "default.go",
        "doc.go",
        "google.go",
        "jwt.go",
        "sdk.go",
    ],
    importpath = "golang.org/x/oauth2/google",
    visibility = ["//visibility:public"],
    deps = [
        "//:oauth2",
        "//authhandler",
        "//google/internal/externalaccount",
        "//internal",
        "//jws",
        "//jwt",
        "@com_google_cloud_go//compute/metadata",
    ],
)

alias(
    name = "go_default_library",
    actual = ":google",
    visibility = ["//visibility:public"],
)

go_test(
    name = "google_test",
    srcs = [
        "example_test.go",
        "google_test.go",
        "jwt_test.go",
        "sdk_test.go",
    ],
    data = glob(["testdata/**"]),
    embed = [":google"],
    deps = [
        "//:oauth2",
        "//jws",
        "//jwt",
    ],
)

Repo B imports Repo A as a Workspace and loads the same Go dependencies - I confirmed this using the --experimental_workspace_rules_log_file flag to dump the archives fetched and compared the sums across executions in both repos. However, the generated BUILD file is:

load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")

go_library(
    name = "google",
    srcs = [
        "appengine.go",
        "appengine_gen2_flex.go",
        "default.go",
        "doc.go",
        "google.go",
        "jwt.go",
        "sdk.go",
    ],
    importpath = "golang.org/x/oauth2/google",
    visibility = ["//visibility:public"],
    deps = [
        "//:oauth2",
        "//authhandler",
        "//google/internal/externalaccount",
        "//internal",
        "//jws",
        "//jwt",
        "@com_google_cloud_go_compute//metadata:go_default_library",
    ],
)

alias(
    name = "go_default_library",
    actual = ":google",
    visibility = ["//visibility:public"],
)

go_test(
    name = "google_test",
    srcs = [
        "example_test.go",
        "google_test.go",
        "jwt_test.go",
        "sdk_test.go",
    ],
    data = glob(["testdata/**"]),
    embed = [":google"],
    deps = [
        "//:oauth2",
        "//jws",
        "//jwt",
    ],
)

...which lists the ref to the nonexistent @com_google_cloud_go_compute workspace.

So it seems this behavior is timing/order-dependent as well, since we see this reliably in one repo but not another with the same Go deps?

[minor-fixes]

More context: compute library was indeed made a module yesterday: https://github.com/googleapis/google-cloud-go/pull/5273

[minor-fixes]

Found a workaround - add a build_directives attribute to the go_repository rules that depend on cloud.google.com/go/compute, like:

go_repository(
    name = "org_golang_x_oauth2",
    importpath = "golang.org/x/oauth2",
    sum = "h1:pkQiBZBvdos9qq4wBAHqlzuZHEXo07pqV06ef90u1WI=",
    version = "v0.0.0-20210514164344-f6687ab2804c",
    build_directives = [
        "gazelle:resolve go cloud.google.com/go/compute/metadata @com_google_cloud_go//compute/metadata"
    ],
)

In our case, since the go_repository is added by running a macro from another repository, we need to re-declare the go_repository rule with the build_directives attribute as desired, before the macro (as the first entry wins).

[thompsonja]

Thanks for that workaround, it's better than adding com_google_cloud_go_compute to the WORKSPACE as it doesn't require pulling an upgraded version of the compute/metadata library if that isn't desired.

[robfig]

This is the code that resolves an import path to a remote module: https://github.com/bazelbuild/bazel-gazelle/blob/master/repo/remote.go#L354-L368

Testing locally without any Bazel repos defined, only a go.mod specifying an earlier version, I can reproduce the incorrect behavior. It appears to be because we don't take the go.mod into account: https://github.com/bazelbuild/bazel-gazelle/blob/master/repo/remote.go#L409-L438

It runs go get -d $importPath in a temp directory to find the module, so your hypothesis is correct. It was introduced in this commit to fix #972: https://github.com/bazelbuild/bazel-gazelle/commit/f33a560acb1a59561f775f296b857294ad5914d8

I'm not sure why it sets up a temp directory with an empty go.mod instead of running go list in the workspace. Without knowing that, I'm hesitant to change the behavior -- any idea @jayconrod ?

This should probably be fixed, but also I'm surprised that you are seeing this, since Gazelle should know about your existing Bazel repo declarations and use those whenever possible, instead of taking this route. If they are defined in a helper bzl file, they should have a # gazelle:repository_macro directive to tell Gazelle. This should fix your problem and also make Gazelle run more quickly. This is a common thing to miss since it's not immediately obvious and Gazelle doesn't complain.. ideally we could do a better job ensuring that Gazelle is not missing them, but I'm not sure what the right mechanism would be.