numaflow-go icon indicating copy to clipboard operation
numaflow-go copied to clipboard

Published 20 hours ago • verified publisher icon numaproj

chore: centralize proto files

Open whynowy opened this issue 2 years ago • 7 comments

https://github.com/numaproj/numaflow/pull/1402

whynowy avatar Dec 01 '23 23:12 whynowy

Should we completely delete the proto file from here or just indicate in the CI process that it has diverged? Otherwise developing new changes might get tricker as it pulls from the main branch of numaflow.

vigith avatar Dec 01 '23 23:12 vigith

Should we completely delete the proto file from here or just indicate in the CI process that it has diverged? Otherwise developing new changes might get tricker as it pulls from the main branch of numaflow.

Good question.

I believe this would be the right process for any other sdks other than golang - however golang will always be the 1st one if there's any change to the protocols, we need to think about how to make it smooth for golang.

whynowy avatar Dec 02 '23 00:12 whynowy

I imagine the process of introducing proto file changes would be:

  1. A PR in numaflow repo for proto file changes.
  2. During Go sdk development, temporarily points to the numaflow PR branch to download proto files for codegen, and raise a PR;
  3. Merge numaflow PR;
  4. Merge Go sdk PR.
  5. Work on other SDKs implementation.
  6. Add a non-blocking CI to run make proto for all the sdks, to indicate if there's drift on the codegen.

Another option is, use Go SDK repo as the the location for centralized proto files.

@vigith - WDYT?

whynowy avatar Dec 04 '23 20:12 whynowy

I imagine the process of introducing proto file changes would be:

  1. A PR in numaflow repo for proto file changes.
  2. During Go sdk development, temporarily points to the numaflow PR branch to download proto files for codegen, and raise a PR;
  3. Merge numaflow PR;
  4. Merge Go sdk PR.
  5. Work on other SDKs implementation.
  6. Add a non-blocking CI to run make proto for all the sdks, to indicate if there's drift on the codegen.

Sounds good to me.

I think we can make the main in this URL path can be overrideable via ENV var? This will help in fetching the proto file easier for development without editing the script.

vigith avatar Dec 04 '23 23:12 vigith

I imagine the process of introducing proto file changes would be:

  1. A PR in numaflow repo for proto file changes.
  2. During Go sdk development, temporarily points to the numaflow PR branch to download proto files for codegen, and raise a PR;
  3. Merge numaflow PR;
  4. Merge Go sdk PR.
  5. Work on other SDKs implementation.
  6. Add a non-blocking CI to run make proto for all the sdks, to indicate if there's drift on the codegen.

Sounds good to me.

I think we can make the main in this URL path can be overrideable via ENV var? This will help in fetching the proto file easier for development without editing the script.

Makes sense.

whynowy avatar Dec 05 '23 22:12 whynowy

Could we please hold off on merging this pull request for now? I am currently making some changes to the proto files in two of my pending PRs. Merging this right now could create conflicts with those changes.

@yhl25 - can you remove the change request?

whynowy avatar Jan 18 '24 07:01 whynowy

Proto Generate will be green after https://github.com/numaproj/numaflow/pull/1466 is merged.

whynowy avatar Jan 18 '24 07:01 whynowy