Support proto definitions from multiple sources

[dcrodman]

Hi, apologies if this exists already (I couldn't find an issue for it) but I'm trying to configure Console with protos from a couple of different sources and have realized that this doesn't really work.

For example, take a config that looks like this:

kafka:
  protobuf:
    enabled: true
    filesystem:
      enabled: true
      paths:
        - /etc/console/protos
    git:
      enabled: true
      repository:
        url: https://github.com/googleapis/googleapis.git
    mappings:
      - topicName: events
        valueProtoType: my.company.Event

The code here seems to prioritize the protos from the filesystem service and, if you also have a Git mapping like the above, it will quietly overwrite the list of files used by the proto registry with the ones from the filesystem. This effectively means that you can only have one Git source that is all-or-nothing, which may not work for a lot of people. In our case, we'd like to tie in the Google proto APIs in addition to our own internal proto definitions.

It would be really nice to be able to define multiple protobuf sources and have Console read them all and merge the results. If there's a good reason not to do this, I'd suggest mentioning in the docs/config/console.yaml documentation that only one of them may be defined.

[weeco]

Hey @dcrodman , thanks for filing the issue and describing your usecase.

Just to make sure: Are you specififically interesrted in providing proto definitions from two different providers (git / filesystem)?

Today, you would be able to provide a git repo providing all proto definitions and then configuring multiple import paths inside of that git repo, see here: https://docs.redpanda.com/docs/manage/console/protobuf/#github-repository

[dcrodman]

Hi @weeco ,

Sorry for not being clear there, I'm specifically interested in two cases:

• git + filesystem
• git + git (two different repositories)

Support for either one of these scenarios would be great, though the ability to define multiple Git repositories would probably be more generally applicable (and honestly would work better for us). In the situation I initially described, combining both repos in a single Git repository isn't very practical for us since we're interested in a third-party repo as well as a path within our internal one.

A potential workaround would require something like an intermediate repo that either vendored or included the others as periodically updated submodules. Which is probably doable but seems like a little too much of a hack to be desirable.