Generate protobuf content during build

[absoludity]

Description:

While we were developing the kubeapps-apis service, we didn't update the bulid process to generate the protobuf content, but should do so now that we're using and depending on this service.

I had the issue yesterday that running buf generate on master leads to code that won't pass our ts-compile checks. I've documented that separately upstream at https://github.com/stephenh/ts-proto/issues/432 , but had we been generating during our build, we would not have updated the ts-proto module without noticing.

We could now do something like git ignoring the generated content in our local checkouts and generating it in the build, or instead failing the build if the output is different. Either way, we'll want to ensure we use the same version of buf locally as in CI etc. Anyway, needs more discussion, just highlighting for inclusion soon.

[castelblanque]

My experience from other projects is that generated source code is never added to Git. This is, removed from repo and added to gitignore. It will be generated in an early phase of the build, either locally or in CI. If developers can not modify it at all, why having it in the repo? If any change needs to be done to the API, the definition file is changed. For development purposes, it works nice if the IDE can generate everything under /gen automatically. Maven or Gradle did the trick, no idea if we could do that in Kubeapps 😄

[absoludity]

My experience from other projects is that generated source code is never added to Git. This is, removed from repo and added to gitignore.

Yeah, that's the normal approach with generated source code and it is something we can do in Kubeapps as above. GRPC-generated code in go is sometimes (but not always) treated differently because the normal tooling is to go get a go module and expect to be able to use it in your code without having to run any third-party tooling. But I don't think we need to cater for this case anyway, so +1 to remove it from the repo and add locally to git ignore as outlined above. The APIs service is no longer just a proof-of-concept.

[antgamdia]

I do agree. It's important to ensure we are using the same version in every environment.

For the record, currently, when running buf generate we are generating: 1) openapi v2 docs; 2) go backend code; 3) ts frontend code Besides, we manually perform some ulterior manual (and potentially undocumented) tasks:

1. convert the openapi v2 json file to an openapi v3 in yaml, copy the paths and schema sections to the openapi.yaml file under dashboard/public folder
2. no action required
3. go to the dashboard directory and run yarn run prettier (or any equivalent way)

During the generation process, we depend on 1) the buf binary already installed in our systems (which embed protoc) and is out of the scope of our VCS system. and 2) the buf mod deps, which are tracked in git.

So, if we plan to untrack the autogenerated code, I foresee two potential issues:

• In development, we wouldn't be able to ensure every dev is using the same protoc version (for instance, I installed buf using brew so I depend on this pkg repo)
• Even if we modify our code to build the images autogenerating the code, note the official Bitnami images won't get that change unless we manually implement it there as well. So, potentially, deleting our code without ensuring their way also works, will result in build failures when releasing new versions.

In short, I'd go with autogenerating the code in CI build time (to catch that kind of bugs) but I wouldn't remove the tracked code yet.