numaflow
numaflow copied to clipboard
Separate repository for proto files
Problem definition
Currently, we keep the proto files in all the SDKs, if we have to change anything we will need to manually update the proto files in all the SDK repos. Which isn't easy to maintain and may cause version differences.
Proposal
-
Keep all the proto files in the numaflow repo, and all the SDKs must copy and use
*.proto
files during build time. -
Create a separate repository just to store and track all the
*.proto
files (daemon.proto, udfs-related protos and info server data could also be defined as a proto) and keep all the generated code in one place and publish them as libraries.
We don't have a lot of proto files at this point but when the number grows it will be difficult to maintain.
sharing some of the interesting blogs that I found related to managing the proto files https://www.bugsnag.com/blog/libraries-for-grpc-services https://www.crowdstrike.com/blog/improving-performance-and-reliability-of-microservices-communication-with-grpc/ https://medium.com/namely-labs/how-we-build-grpc-services-at-namely-52a3ae9e7c35
Side Note: please also resolve https://github.com/numaproj/numaflow-java/pull/36#discussion_r1167368359 as part of this issue.