Error with buf build on bazel external dependency

[cyrilhuang]

trafficstars

We're using buf to do some protobuf breaking detections. Our repo is developing on bazel. Wewant to first buf build to generate image, then to compare reulsts based on image. In one of our proto_library, we depends on external dependency (which is not available on buf remote registry), and import that .proto file in our own code. This proto definition can be found in bazel-out/../../../external/riegeli. bazel will call protoc -I to automatically fix compiling issue.

However, under buf, we cannot add that include path to buf cli/buf.yaml. One official solution is that to use buf.work.yaml. However, in our repo, all proto files are on root, and . is not allowed in buf.work.yaml, therefore creating a buf.yaml in riegeli seems not possible here. example.proto:5:8:read riegeli/records/records_metadata.proto: file does not exist

One temporary solution is to copy that directory into our root directory, then call buf build will work. But I'm wondering what might be the official way to resolve it.

buf version: 1.29.0

[bufdev]

You'll have to provide us a specific, reproducible example, and then we'd be happy to investigate. Let us know!

[cyrilhuang]

@bufdev I'll given a minimum structure of our codebase, please help take a look, thanks In the following example, we can run bazel build //b:proto, since bazel downloads external dependency to local, and add include path to corresponding protoc arg

WORKSPACE

workspace(name = "example")

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
    name = "riegeli",
    commit = "904c0c263b8632265103f0066c168a92c7713b07",
    remote = "ssh://[email protected]:google/riegeli.git",
)

a/BUILD

package(default_visibility = ["//visibility:public"])

proto_library(
    name = "config",
    srcs = ["config.proto"],
)

a/config.proto

syntax = "proto3";
package a;
message config {}

b/BUILD

package(default_visibility = ["//visibility:public"])

proto_library(
    name = "config",
    srcs = ["config.proto"],
    deps = [
        "//a:config",
        "@riegeli//riegeli/records:records_metadata_proto",
    ]
)

b/config.proto

syntax = "proto3";
package b;

import "a/config.proto";
import "riegeli/records/records_metadata.proto";

[bufdev]

What we're looking for:

• The code, in a consumable manner such as a temporary github repository, that we can inspect and build ourselves
• What you see
• What you expected to see

If you can give us something that we can run ourselves, we're happy to look into it.

[cyrilhuang]

@bufdev I put the above example in https://github.com/cyrilhuang/buf_example/tree/main,

In this repo, we can build all proto files with bazel by running:

bazel build //...

However, when running buf, build failed due to cannot find external files:

buf build --disable-symlinks
b/config.proto:5:8:read riegeli/records/records_metadata.proto: file does not exist

The mnissing file can be found in bazel-buf_example/external/riegeli/, which bazel added during compiling proto files (buf_example is the repo name). bazel-buf_example/external/riegeli/riegeli/records/records_metadata.proto

I'm wondering how to deal with external dependency not available on buf remote registry like riegeli, thanks.

bazel version:7.1.1 buf version: 1.30.1

[doriable]

@cyrilhuang Thank you for providing the example repo. Basically, the file you have a dependency on, is in bazel-buf_example/external/riegeli, which is currently being excluded from the builds.excludes where you are excluding the bazel-* directories. You need to give your workspace access to bazel-buf_example/external/riegeli/riegeli, which could be tricky, since you don't want to include everything from your bazel cache.

I've mocked up an example below that would work using your example repo:

1. First I moved your protos into a proto directory:

$ tree
.
├── MODULE.bazel
├── MODULE.bazel.lock
├── WORKSPACE
├── bazel-bin -> ...
├── bazel-buf_example -> ...
├── bazel-out -> ...
├── bazel-testlogs -> ...
├── buf.lock
├── buf.yaml
└── proto
    ├── a
    │   ├── BUILD
    │   └── config.proto
    └── b
        ├── BUILD
        └── config.proto

8 directories, 9 files

2. Then I created a workspace that incorporates your proto definitions and the riegeli dependency, managed by bazel. I used our v2 workspace configuration to do this:

$ cat buf.yaml
version: v2
modules:
  - path: proto
  - path: bazel-buf_example/external/riegeli
    excludes:
      - bazel-buf_example/external/riegeli/python
deps:
  - buf.build/googleapis/googleapis
breaking:
  use:
    - PACKAGE
lint:
  use:
    - DEFAULT

What this basically configures is a workspace with the desired proto definitions. In the case of riegeli, I had to exclude the python directory, since it repeated the records_metadata.proto file. I updated the buf.lock using buf dep update, and now it all works.

Basically, in order to get access to dependencies that you cannot get through the BSR, you'll need to incorporate the local files as a part of your workspace, otherwise buf build will not be able to "see" them. Hope this helps!

We have documentation for configuring your module and workspace, available here: https://buf.build/docs/configuration/v2/buf-yaml

[doriable]

Closing this for now since an answer to the behaviour and how to work with it has been provided above. Please feel free to re-open if there are additional questions or concerns!