bufbuild
`buf breaking`: `google/api/field_behavior.proto: does not exist` while build works
We are currently using buf for generation only, but looking into adding it for breaking as well. However, having some issues.
Repo: https://github.com/istio/api
$ buf breaking --against '.git#branch=master' --path type
type/v1beta1/selector.proto:16:8:google/api/field_behavior.proto: does not exist
However, building and passing the result works:
buf build -o proto.bin; buf breaking --against proto.bin
With verbose:
$ buf breaking --against '.git#branch=master' --path type --verbose
DEBUG get_ref {"duration": "58.392µs"}
DEBUG get_config {"duration": "371.953µs"}
DEBUG get_module_config {"duration": "481.822µs"}
DEBUG parse {"duration": "7.205419ms"}
DEBUG get_image {"duration": "14.422µs"}
DEBUG build {"duration": "58.960046ms"}
DEBUG build_module {"duration": "58.973422ms"}
DEBUG get_ref {"duration": "24.379µs"}
DEBUG git_clone_to_bucket_copy {"duration": "22.369833ms"}
DEBUG git_clone_to_bucket {"duration": "500.536051ms"}
DEBUG get_config {"duration": "296.741µs"}
DEBUG get_module_config {"duration": "501.223975ms"}
DEBUG parse {"duration": "240.609µs"}
DEBUG build {"duration": "505.528µs"}
DEBUG build_module {"duration": "514.043µs"}
type/v1beta1/selector.proto:16:8:google/api/field_behavior.proto: does not exist
DEBUG command {"duration": "561.499345ms"}
The two commands you've shown might not be equivalent. In the first case you're checking against the module built at .git#branch=master, whereas in the second command you're building a proto.bin from the current directory. Did you make sure to check out master to build proto.bin?
If so, you can continue to narrow in on the problem by using buf's introspection features (with jq). Try building each reference independently and checking what files you have available.
# List the files in the Image on the local master branch.
$ buf build .git#branch=master -o -format=json | jq '.file[] | .name'
# List the files in the Image defined in the current directory.
$ buf build . -o -format=json | jq '.file[] | .name'
I was on master - One line reproducer: git clone https://github.com/istio/api; cd api; buf breaking --against '.git#branch=master' --path type
$ buf build .git#branch=master -o -format=json | jq '.file[] | .name'
authentication/v1alpha1/policy.proto:23:8:google/api/field_behavior.proto: does not exist
mcp/v1alpha1/mcp.proto:21:8:google/rpc/status.proto: does not exist
mcp/v1alpha1/metadata.proto:19:8:gogoproto/gogo.proto: does not exist
mesh/v1alpha1/network.proto:19:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/destination_rule.proto:16:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/envoy_filter.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/gateway.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/service_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/sidecar.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/virtual_service.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/workload_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/workload_group.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/destination_rule.proto:16:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/gateway.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/service_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/sidecar.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/virtual_service.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/workload_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/workload_group.proto:17:8:google/api/field_behavior.proto: does not exist
security/v1beta1/authorization_policy.proto:16:8:google/api/field_behavior.proto: does not exist
security/v1beta1/jwt.proto:16:8:google/api/field_behavior.proto: does not exist
type/v1beta1/selector.proto:16:8:google/api/field_behavior.proto: does not exist
Had to tweak the command but:
$ buf build -o f.json ; cat f.json| jq '.file[] | .name'
"google/protobuf/struct.proto"
"analysis/v1alpha1/message.proto"
"google/protobuf/descriptor.proto"
"google/api/field_behavior.proto"
"authentication/v1alpha1/policy.proto"
"envoy/config/filter/http/alpn/v2alpha1/config.proto"
"envoy/config/filter/http/authn/v2alpha1/config.proto"
"google/protobuf/duration.proto"
"envoy/config/filter/http/jwt_auth/v2alpha1/config.proto"
"envoy/config/filter/network/metadata_exchange/metadata_exchange.proto"
"envoy/config/filter/network/tcp_cluster_rewrite/v2alpha1/config.proto"
"google/protobuf/wrappers.proto"
"envoy/extensions/stackdriver/config/v1alpha1/config.proto"
"envoy/extensions/stats/config.proto"
"google/protobuf/any.proto"
"type/v1beta1/selector.proto"
"extensions/v1alpha1/wasm.proto"
"google/rpc/status.proto"
"gogoproto/gogo.proto"
"google/protobuf/timestamp.proto"
"mcp/v1alpha1/metadata.proto"
"mcp/v1alpha1/resource.proto"
"mcp/v1alpha1/mcp.proto"
"networking/v1alpha3/virtual_service.proto"
"networking/v1alpha3/destination_rule.proto"
"networking/v1alpha3/workload_entry.proto"
"networking/v1alpha3/workload_group.proto"
"networking/v1beta1/proxy_config.proto"
"mesh/v1alpha1/proxy.proto"
"k8s.io/apimachinery/pkg/runtime/generated.proto"
"k8s.io/apimachinery/pkg/runtime/schema/generated.proto"
"k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto"
"mesh/v1alpha1/config.proto"
"mesh/v1alpha1/network.proto"
"meta/v1alpha1/status.proto"
"networking/v1alpha3/gateway.proto"
"networking/v1alpha3/sidecar.proto"
"networking/v1alpha3/envoy_filter.proto"
"networking/v1alpha3/service_entry.proto"
"networking/v1beta1/virtual_service.proto"
"networking/v1beta1/destination_rule.proto"
"networking/v1beta1/gateway.proto"
"networking/v1beta1/sidecar.proto"
"networking/v1beta1/workload_entry.proto"
"networking/v1beta1/service_entry.proto"
"networking/v1beta1/workload_group.proto"
"k8s.io/apimachinery/pkg/util/intstr/generated.proto"
"k8s.io/apimachinery/pkg/api/resource/generated.proto"
"k8s.io/api/core/v1/generated.proto"
"operator/v1alpha1/operator.proto"
"security/v1alpha1/ca.proto"
"security/v1beta1/authorization_policy.proto"
"security/v1beta1/jwt.proto"
"security/v1beta1/peer_authentication.proto"
"security/v1beta1/request_authentication.proto"
"stability/level.proto"
"telemetry/v1alpha1/telemetry.proto"
It looks like you're using symlinks in your repository, which aren't supported for .git references. Your buf build command succeeds with the local sources because symlinks are supported for local source references.
But now that I'm taking a closer look, it looks like you're really trying to battle the buf compiler into getting things working. Plus, you're using the v1beta1 configuration version in your buf.yaml. I'll play around with this a bit and see if I can get it into a good spot for you.
All right, there's a few things to unravel here. Your current setup has a structure that makes it hard to shift around all at once, so I'll outline the ideal approach instead of trying to do it myself. It'll require that you move around your .proto sources, and define new buf.gen.yaml templates to preserve the old import paths you were using before.
Ideally you can capture your concept of common-protos with a buf.work.yaml (described here). The only problem with that is that all of your other core .proto sources would need to be in a separate directory (we recommend a proto directory).
With this, you could define a buf.work.yaml like so:
# buf.work.yaml
version: v1
directories:
- common-protos
- proto
Now, you could get rid of a lot of the content in common-protos and replace them with remote dependencies. We have the gogo, googleapis modules available to you, and the well-known types (your google/protobuf/... files) are automatically handled for you. Plus, it looks like you're copying some envoy files outside of common-protos, too. We can use the remote envoy module for those, too. You can establish those dependencies with the buf.yaml moved into the proto directory like so:
# proto/buf.yaml
version: v1
deps:
- buf.build/envoyproxy/envoy
- buf.build/gogo/protobuf
- buf.build/googleapis/googleapis
lint:
allow_comment_ignores: true
use:
- BASIC
except:
- FIELD_LOWER_SNAKE_CASE
- PACKAGE_DIRECTORY_MATCH
With this buf.yaml, you can run buf mod update to create a buf.lock manifest which includes those dependencies in your build.
That leaves just your k8s.io and istio files in the common-protos directory, all of which get consolidated into your build because of the buf.work.yaml.
So far this means your repository would look like the following:
├── buf.gen.yaml
├── buf.work.yaml
├── common-protos
│ ├── istio.io
│ │ └── extensions
│ │ └── field_rules.proto
│ └── k8s.io
│ ├── ...
├── proto
│ ├── analysis
│ │ └── v1alpha1
│ │ └── message.proto
│ ├── authentication
│ │ └── v1alpha1
│ │ └── policy.proto
│ ├── buf.lock
│ ├── buf.yaml
│ ├── extensions
│ │ └── v1alpha1
│ │ └── wasm.proto
│ ├── mcp
│ │ └── v1alpha1
│ │ ├── mcp.proto
│ │ ├── metadata.proto
│ │ └── resource.proto
│ ├── mesh
│ │ └── v1alpha1
│ │ ├── config.proto
│ │ ├── network.proto
│ │ └── proxy.proto
│ ├── meta
│ │ └── v1alpha1
│ │ └── status.proto
│ ├── networking
│ │ ├── v1alpha3
│ │ │ ├── destination_rule.proto
│ │ │ ├── envoy_filter.proto
│ │ │ ├── gateway.proto
│ │ │ ├── service_entry.proto
│ │ │ ├── sidecar.proto
│ │ │ ├── virtual_service.proto
│ │ │ ├── workload_entry.proto
│ │ │ └── workload_group.proto
│ │ └── v1beta1
│ │ ├── destination_rule.proto
│ │ ├── gateway.proto
│ │ ├── proxy_config.proto
│ │ ├── service_entry.proto
│ │ ├── sidecar.proto
│ │ ├── virtual_service.proto
│ │ ├── workload_entry.proto
│ │ └── workload_group.proto
│ ├── operator
│ │ └── v1alpha1
│ │ └── operator.proto
│ ├── security
│ │ ├── v1alpha1
│ │ │ └── ca.proto
│ │ └── v1beta1
│ │ ├── authorization_policy.proto
│ │ ├── jwt.proto
│ │ ├── peer_authentication.proto
│ │ └── request_authentication.proto
│ ├── stability
│ │ └── level.proto
│ ├── telemetry
│ │ └── v1alpha1
│ │ └── telemetry.proto
│ └── type
│ └── v1beta1
│ └── selector.proto
Now, the problem is that you've generated your Go code (and other artifacts) alongside your .proto definitions. This is generally a bad idea for exactly the reason we're seeing here. Now that we want to move the .proto definitions around to accommodate a solid structure (using a buf.work.yaml, etc), you need to generate those files into the same place they were, or choose another generation output directory and adjust all of your import paths + scripts that depend on them.
You should be able to use a combination of buf.gen.yaml templates to do this (like the one you have now), but this will require some extra domain expertise in your repository that you'll have more than I do.
That was a mouthful, but all of this information is outlined in the tour, and other reference material outlined at https://docs.buf.build. I'd strongly encourage you to complete the tour to see how all of these pieces are arranged. After that, you can restructure your .proto files so that they aren't all defined from the root of your repository and you'll be good to go. It'll take a bit of work, but it'll make it much easier to separate your API definitions from the generated code they produce.
I'll close this out as-is. Hopefully my previous comment helped clear up the problem and solution!