Support Protobuf Editions

[deepakm37]

🚀 Feature

Protobuf is replacing the current syntax proto2 & proto3 with Editions. All future releases will be new Editions. Edition 2023 is already released as well.

The proto plugin for grpc-gateway doesn't yet support Editions. It will be great to add this support as users switch to the new syntax.

[johanbrandhorst]

Hi, thanks for this issue. I agree that it is important that the gateway stays up to date with the protobuf ecosystem as it evolves. Have you tried using our generators with the new editions syntax? How does it fail, if at all? I was under the impression that editions are optional and may not require any changes to generators. Are there any new features you anticipate will affect the gateway?

[deepakm37]

I have tried using the current plugin (in Bazel through rules_proto_grpc_gatway) and encountered failures. As plugin doesn't explicitly notify that it supports editions, proto compilations fails. Here is a sample error message from Bazel:

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
downloaded: invalid FileDescriptorProto "service.proto": proto: invalid syntax: "editions"
--plugin_out: protoc-gen-plugin: Plugin failed with status code 1.

You're right that adding editions may not require any changes from gateway perspective. I believe if the plugin can just notify that it supports editions, without any functional changes, it should be able to handle protos in the new syntax.

[johanbrandhorst]

Hm, I wonder if this is just a matter of updating our dependencies. In any case, thanks for the description. Would you be able to help contribute this?

[deepakm37]

Sure thing. I will take a look at the gateway plugin source. I will get back to you when I have a PR. In the mean time, please share any details that may be helpful to me, given that this would be my first contribution here. Thanks!

[johanbrandhorst]

The logic is all in protoc-gen-openapiv2 and protoc-gen-grpc-gateway respectively, it shouldn't be too bad to dive into, hopefully!

[deepakm37]

I took a look at the plugin and included EDITION 2023 in its supported features. Unfortunately that didn't work i.e. protoc didn't complain but the generated code was not valid. Up on digging deeper, it looks like the templating and template functions need to be updated to support field presence, among other things.

I also noticed that the code generator, as it is, didn't quite work for proto2 -- for instance, using syntax = "proto2" (and making all fields optional) in //examples/internal/proto/examplepb/echo_service.proto generated invalid code as observed when building //examples/internal/proto/examplepb:examplepb_go_proto.

Let me know if I'm understanding this incorrectly. I was wrong in underestimating the changes needed to add Editions support. I will revisit this later when I have more cycles.

[johanbrandhorst]

Thanks for taking a look. I'm surprised to hear that proto2 syntax is causing issues, I wouldn't expect that to be the case. This might be the right time for at least the protoc-gen-grpc-gateway generator to move to something like https://github.com/protocolbuffers/protobuf-go/tree/master/compiler/protogen, though I don't know how much work that would be.

[yulin-li]

Any progress on this?

[johanbrandhorst]

Nope. Would you be willing to contribute support for this?

[Dogacel]

I wonder how many people are using editions in production right now. Any idea why there is so little support? I want to contribute however I need to get familiar with Go/Protobuf ecosystem, I am mostly coming form Java / Kotlin.

[mkruskal-google]

FYI, we have some information on adding editions support for plugins: https://protobuf.dev/editions/implementation/

If you support proto2 and proto3 already it's mostly a matter of plumbing features everywhere, making sure they get resolved, and removing any explicit checks of syntax. The only actual new functionality added in edition 2023 is that group-encoding can be specified on any message field instead of just special ones with matching names.

That or relying on descriptors from go-protobuf which already supports editions, idk how much custom support you'd need on top of that

[kop]

I created a PR that is addressing this issue: https://github.com/grpc-ecosystem/grpc-gateway/pull/5723

[printfn]

Would it be possible to get a new release with this feature included?

[johanbrandhorst]

https://github.com/grpc-ecosystem/grpc-gateway/releases/tag/v2.27.3 done :)