Simplify building subsets of a workspace

[jeffsawatzky]

I have a repo layout similar to the following:

apis/
   service_a/
      service_a/
         - service.proto
      - buf.yaml
   service_b/
      service_b/
         - service.proto
      - buf.yaml
   service_common/
      service_common/
         - common.proto
      - buf.yaml
   - buf.gen.yaml
   - buf.work.yaml

When it comes time to build binary descriptor sets to use in envoy it would be nice to be able to do something like:

buf build -o service_a.bin --as-file-descriptor-set --path service_a --path service_common

So that we can create individual files for service_a and service_b that only includes the service specific and common definitions.

However, that errors with:

Failure: path "service_a" is equal to workspace directory "service_a" defined in "buf.work.yaml"

So I need to end up running it with:

buf build -o service_a.bin --as-file-descriptor-set --path service_a/service_a --path service_common/service_common

Note the repetition in the --path values.

Another option would be to allow buf to accept multiple input directories like:

buf build service_a service_common -o service_a.bin --as-file-descriptor-set

[bufdev]

Yea fair point. The error message is basically saying "you are using --path for something we didn't intend", the thing we didn't intend being you're using --path to specify a module path, not a file or directory path. I think the way you'd want to do this is your second option, taking multiple inputs. This actually might not be that bad, and be backwards compatible, but could have some ugly downstream effects, we'll take a look.

[bufdev]

@jeffsawatzky can you give this a try?

go install github.com/bufbuild/buf/cmd/buf@multiple-inputs
buf build service_a service_common -o service_a.bin --as-file-descriptor-set

The code is https://github.com/bufbuild/buf/compare/multiple-inputs?expand=1 - as you can see there, there's a lot of TODOS. I can't promise we could resolve all of those quickly and in a timely manner, and if we did this, we'd also want to make it consistent across the whole codebase. This is a way of saying "no promises we can do this in the short term". But I want to see if this is generally what you're looking for (and bonus points if you can check out what happens with the --path and --exclude-path flags in this scenario).

[jeffsawatzky]

@bufdev I tried doing as you requested, but I am running into an error. I am not a go developer, so I am not sure exactly the correct process, but this is what I have:

1. I have a docker container made FROM python:3.9-slim-bullseye
2. I export GOBIN=/usr/local/bin
3. I install golang with apt-get install golang
4. I run your command and I get the following error

package github.com/bufbuild/buf/cmd/buf@multiple-inputs: cannot use path@version syntax in GOPATH mode

[bufdev]

You're likely using an old version of Go. Use golang:1.20.5-bullseye https://hub.docker.com/_/golang

[jeffsawatzky]

@bufdev, yeah that was it. I was able to test this, and the change seems to do what I would like.

[bufdev]

We reviewed this a bit more from a design perspective, and I think we're going to close this at this point as infeasible given our priorities. Apologies for that.