Trouble passing plugin flags to protoc via go_proto_compiler

[ZacxDev]

What version of rules_go are you using?

v0.24.3

What version of gazelle are you using?

v0.22.2

What version of Bazel are you using?

Build label: 3.5.1-homebrew Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar Build time: Thu Oct 1 17:34:08 2020 (1601573648) Build timestamp: 1601573648 Build timestamp as int: 1601573648

Does this issue reproduce with the latest releases of all the above?

yes

What operating system and processor architecture are you using?

macOS x86

Any other potentially useful information about your toolchain?

What did you do?

Attempting to use a protoc plugin like so:

go_proto_compiler(
       name = "struct_transformer",
       options = [
          "struct-transformer_out=package=transform,helper-package=helpers:.",
           "go_out=Moptions/annotations.proto=github.com/ZacxDev/protoc-gen-struct-transformer/options",
       ],
       plugin = "@com_github_ZacxDev_protoc_gen_struct_transformer//:protoc-gen-struct-transformer",
       suffix = ".transformer.pb.go",
   )

What did you expect to see?

successful build with transformer functions generated

What did you see instead?

2020/11/09 16:11:37 no such flag -struct-transformer_out
--struct-transformer_out: protoc-gen-struct-transformer: Plugin failed with status code 1.

Sorry if this is a simple case of user error, I am very new to Bazel. I have been stuck on this for about a week now and am getting close to giving up, this issue is my only hope!

I've tried playing around with different flag formats but nothing seems to take. I really just need to pass these options as flags to protoc, from what I've read go_proto_compiler/options is the right place but I really am not sure.

Thanks for any help <3

[jayconrod]

The options flags here don't seem quite right to me, but I'm not sure what the right values would be.

go_proto_compiler invokes protoc with an argument like:

--foo_out=opt1,opt2:dir

where

• foo is the name of the plugin (without the protoc-gen- prefix or .exe suffix on Windows)
• opt1,opt2 are the options, joined with commas
• dir is a temporary directory where the outputs are written

[ZacxDev]

Thanks for the quick response. That definitely helps.

Based on what you said, I've modified my BUILD file to have this:

go_proto_compiler(
    name = "struct-transformer",
    options = [
        "package=transform",
        "helper-package=helpers",
    ],
    plugin = "@com_github_ZacxDev_protoc_gen_struct_transformer//:protoc-gen-struct-transformer",
    suffix = ".transformer.pb.go",
)

go_proto_compiler(
    name = "go-struct-transformer",
    options = [
        "Moptions/annotations.proto=github.com/ZacxDev/protoc-gen-struct-transformer/options",
    ],
)

The .pb.go file generates fine, but the transformer.pb.go file comes out with:

// +build ignore

package ignore

Any ideas?

[jayconrod]

I think that can happen if a plugin doesn't generate any output files. go_proto_compiler will write a file like that in the absence of anything else.

[ZacxDev]

I think I am closer to getting this working. In the go_proto_compiler rule for the protoc plugin I've set valid_archive to false and it now builds correctly. That being said, I'm unsure how to use it in my go_library. If I include it as a dep, I get this error:

ERROR: /Users/zacharylowden/workspace/mobl-dev/packages/user-api/rpci/BUILD.bazel:3:11: in go_library rule //packages/user-api/rpci:rpci:
Traceback (most recent call last):
        File "/private/var/tmp/_bazel_zacharylowden/3ad95b156d64321b76ac378b18d2001c/external/io_bazel_rules_go/go/private/rules/library.bzl", line 38, column 25, in _go_library_impl
                archive = go.archive(go, source)
        File "/private/var/tmp/_bazel_zacharylowden/3ad95b156d64321b76ac378b18d2001c/external/io_bazel_rules_go/go/private/actions/archive.bzl", line 65, column 26, in emit_archive
                direct = [get_archive(dep) for dep in source.deps]
        File "/private/var/tmp/_bazel_zacharylowden/3ad95b156d64321b76ac378b18d2001c/external/io_bazel_rules_go/go/private/providers.bzl", line 88, column 15, in get_archive
                return dep[GoArchive]
Error: <target //packages/user-api:user_go_proto_transform> (rule 'go_proto_library') doesn't contain declared provider 'GoArchive'
ERROR: Analysis of target '//packages/user-api:user-api' failed; build aborted: Analysis of target '//packages/user-api/rpci:rpci' failed

If I try to embed it I get an error saying "tried to write the same file twice". Here is my BUILD.bazel file for reference:

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
load("@rules_proto//proto:defs.bzl", "proto_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")
load("@io_bazel_rules_go//proto:compiler.bzl", "go_proto_compiler")
load(
    "//:wkt.bzl",
    "PROTO_RUNTIME_DEPS",
    "WELL_KNOWN_TYPE_RULES",
)

go_proto_compiler(
    name = "struct-transformer",
    options = [
        "package=transform",
        "helper-package=helpers",
        "goimports=true",
    ],
    plugin = "@com_github_ZacxDev_protoc_gen_struct_transformer//:protoc-gen-struct-transformer",
    suffix = "_transform.pb.go",
    deps = [
        ":user_go_proto",
        "//packages/user-api/helpers",
        "//packages/user-api/model",
    ],
    valid_archive = False,
)

go_proto_compiler(
    name = "go_grpc_struct_transformer",
    options = [
        "Moptions/annotations.proto=github.com/ZacxDev/protoc-gen-struct-transformer/options",
        "plugins=grpc",
    ],
    deps = PROTO_RUNTIME_DEPS + WELL_KNOWN_TYPE_RULES.values() + [
        "@org_golang_google_grpc//:go_default_library",
        "@org_golang_google_grpc//codes:go_default_library",
        "@org_golang_google_grpc//status:go_default_library",
        "@org_golang_x_net//context:go_default_library",
    ],
)

proto_library(
    name = "user_proto",
    srcs = ["UserService.proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//shared/protos:common_proto",
        "@com_github_ZacxDev_protoc_gen_struct_transformer//options:options_proto"
    ],
)

go_proto_library(
    name = "user_go_proto",
    compilers = [
        ":go_grpc_struct_transformer",
    ],
    importpath = "gitlab.com/mobl-dev/packages/user-api/pb",
    protos = [":user_proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//shared/protos:common_go_proto",
        "@com_github_ZacxDev_protoc_gen_struct_transformer//options:options_go_proto",
    ],
)

go_proto_library(
    name = "user_go_proto_transform",
    compilers = [
        ":struct-transformer",
    ],
    importpath = "gitlab.com/mobl-dev/packages/user-api/pb/transform",
    protos = [":user_proto"],
    visibility = ["//visibility:public"],
    deps = [
        "//packages/user-api/helpers",
        "//packages/user-api/model",
    ],
)

go_library(
    name = "user-api_lib",
    srcs = ["main.go"],
    importpath = "gitlab.com/mobl-dev/packages/user-api",
    visibility = ["//visibility:private"],
    deps = [
        "@com_github_bugsnag_bugsnag_go//:bugsnag-go",
        "@com_github_googlecloudplatform_opentelemetry_operations_go_exporter_trace//:trace",
        "@com_github_grpc_ecosystem_go_grpc_middleware//:go-grpc-middleware",
        "@com_github_grpc_ecosystem_go_grpc_middleware//validator",
        "@dev_unknwon_clog_v2//:clog",
        "@io_opentelemetry_go_contrib_instrumentation_google_golang_org_grpc//:grpc",
        "@io_opentelemetry_go_otel//api/global",
        "@io_opentelemetry_go_otel_sdk//trace",
        "@org_golang_google_grpc//:go_default_library",
        ":user_go_proto",
        "//packages/user-api/rpci",
    ],
)

go_binary(
    name = "user-api",
    embed = [":user-api_lib"],
    visibility = ["//visibility:public"],
)

The plugin works fine with protoc so it should be able to work with Bazel if I can figure out the right rules (I am new to Bazel so getting even this far has been a bumpy road).

Thanks for any help! <3

[jayconrod]

Setting valid_archive to False means that compiler won't produce a complete, buildable archive. Specifically, it means that a go_proto_library target that uses only that compiler won't provide GoArchive, meaning it can't be used as a dependency.

If there are static files in the same package that "complete" the package, then the go_proto_library can be embedded into a go_library, and the files will be compiled together. This is rare though: generated proto code should be in a separate package from code that uses it.

If the compiler is meant to be used together with other compilers, and at least one of them does not set valid_archive = False, then the go_proto_library will provide GoArchive and can be used as a dependency.

[ZacxDev]

Ok, with that in mind then I don't think valid_archive should be false. When I remove valid_archive and try to compile user_go_proto_transform, I get the following error:

ERROR: /Users/zacharylowden/workspace/mobl-dev/packages/user-api/BUILD.bazel:65:17: Generating into bazel-out/darwin-fastbuild/bin/packages/user-api/user_go_proto_transform_/gitlab.com/mobl-dev/packages/user-api/pb/transform failed (Exit 1): go-protoc-bin failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/io_bazel_rules_go/go/tools/builders/go-protoc-bin_/go-protoc-bin -protoc bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc ... (remaining 25 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox go-protoc-bin failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/io_bazel_rules_go/go/tools/builders/go-protoc-bin_/go-protoc-bin -protoc bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc ... (remaining 25 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
user-api: Tried to write the same file twice.
2020/11/12 10:05:32 error running protoc: exit status 1
Target //packages/user-api:user_go_proto_transform failed to build
Use --verbose_failures to see the command lines of failed build steps.

I'm not sure how to debug the user-api: Tried to write the same file twice. error, this plugin works fine on its own with protoc.

[jamesthompson]

@ZacxDev did you ever manage to fix this issue by chance?

I'm seeing the following:

// +build ignore

package ignore

As the output of my plugin as well, even though I've validated that the proto compiler plugin is indeed pushing the correct output to stdout. I've tried setting the options = [ "$NAME_out=:." ] argument, where my plugin binary is called $NAME in the go_proto_compiler call to no avail as well. I'm stumped.

I'm trying to make use of proto-gen-star as it looks really excellent.

[jamesthompson]

tl;dr:

The go_proto_compiler rule requires the following, I managed to get this working:

    import_path_option = True,
    plugin = ":protoc-gen-jsonify",
    suffix = ".pb.json.go", # the suffix format must follow this .pb.$x.go format
    options = ["plugins=jsonify"], # This plugins option must be specified to tell protoc what it needs to do

e.g.

load("@io_bazel_rules_go//go:def.bzl", "go_binary")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")
load("@io_bazel_rules_go//proto:compiler.bzl", "go_proto_compiler")
load("@rules_proto//proto:defs.bzl", "proto_library")

go_binary(
    name = "protoc-gen-jsonify", # see https://github.com/lyft/protoc-gen-star/tree/master/testdata/protoc-gen-example, I just copied the files over from here for testing
    srcs = glob(["*.go"]),
    importpath = "main",
    pure = "on",
    visibility = ["//visibility:private"],
    deps = [
        "@com_github_lyft_protoc_gen_star//:go_default_library",
        "@com_github_lyft_protoc_gen_star//lang/go:go_default_library",
    ],
)

go_proto_compiler(
    name = "jsonify_compiler",
    import_path_option = True,
    plugin = ":protoc-gen-jsonify",
    suffix = ".pb.json.go",
    options = ["plugins=jsonify"], # !!This is the magic incantation that you need to get the plugin to actually write out!!
    deps = [ # I just copied these over as deps from the regular proto compiler from rules_go
        "@com_github_golang_protobuf//proto:go_default_library",
        "@org_golang_google_protobuf//proto:go_default_library",
        "@org_golang_google_protobuf//reflect/protoreflect:go_default_library",
        "@org_golang_google_protobuf//runtime/protoiface:go_default_library",
        "@org_golang_google_protobuf//runtime/protoimpl:go_default_library",
        "@org_golang_google_protobuf//types/descriptorpb",
        "@org_golang_google_protobuf//types/known/anypb",
        "@org_golang_google_protobuf//types/known/apipb",
        "@org_golang_google_protobuf//types/known/durationpb",
        "@org_golang_google_protobuf//types/known/emptypb",
        "@org_golang_google_protobuf//types/known/fieldmaskpb",
        "@org_golang_google_protobuf//types/known/sourcecontextpb",
        "@org_golang_google_protobuf//types/known/structpb",
        "@org_golang_google_protobuf//types/known/timestamppb",
        "@org_golang_google_protobuf//types/known/typepb",
        "@org_golang_google_protobuf//types/known/wrapperspb",
        "@org_golang_google_protobuf//types/pluginpb",
    ],
)

proto_library(
    name = "proto",
    srcs = [
        "test.proto", # just a test protobuf file I created
    ],
    visibility = ["//visibility:public"],
    deps = [
    ],
)

go_proto_library(
    name = "test_generate",
    compilers = [
        ":jsonify_compiler",
    ],
    importpath = "testpb",
    proto = ":proto",
    visibility = ["//visibility:public"],
    deps = [
        "@com_github_golang_protobuf//jsonpb:go_default_library",
    ]
)