GoGo Protobuf looking for new ownership

[awalterschulze]

I write to you on behalf of the GoGo Protobuf project, which has long been supported by a small group of dedicated developers taking time out of their lives to keep the project up to date and add new features, to the benefit of all users. Unfortunately, the personal circumstances of the maintainers have changed and we are no longer able to keep up with the issues and feature requests that naturally crop up for a popular open source project.

In particular, the recent golang/protobuf release 1.4.x (AKA APIv2: https://blog.golang.org/protobuf-apiv2), has created a big chunk of work required to be compatible with the new world of Go protobufs. While longer term this work will likely make gogo/protobuf much more compatible with the rest of the ecosystem, there is currently no one to take on this piece of work, and users are living in uncertainty around the future of the project.

We are therefore reaching out to industry users such as yourself to ask for help with maintaining the project. Without external help, it is likely that the GoGo Protobuf project will be discontinued. We know there are many organisations out there that use the project, so we're hoping that one or more of them would step up to take on this maintenance.

Please let us know if you are a user that can take on (co)ownership of this project. It is as simple as commenting on this issue.

Sincerely,

@awalterschulze and @johanbrandhorst

From Walter:

I want to thank @jmarais and @donaldgraham for doing a great job stepping up to take on maintenance of GoGo Protobuf over the past 2 years. I want to also thank @johanbrandhorst for developing and maintaining the GRPC interoperability. I am very lucky and proud to call you all my friends.

[alexanderbez]

I would really hate to see this project die and would love to see parity with the new Proto APIv2. I would like to help contribute as much as I can. Do you think before the existing core contributors move on, can create a milestone or set of issues needing to be completed to be compatible with the latest v2 updates?

[tac0turtle]

I share the same sentiment. I would like to contribute and help organize gogoproto. That being said I am unfamiliar with the inner workings of gogoproto and I am a junior developer, but I am willing to learn and work with others to help this project live on.

[johanbrandhorst]

The first task is to identity the current incompatibilities with APIv2. Just a quick look at the issue tracker gives me:

• https://github.com/gogo/protobuf/issues/686
• https://github.com/gogo/protobuf/issues/678

There will probably be more, but it would be good to first get an idea of the exact shortcomings and necessary fixes. Assuming no bigger org steps up to take over, I think the short term goal is 1.4.x compatibility. If ownership can be transferred, there could be talk of a GoGo protobuf APIv2.

[dhiemaz]

Would love to take part of this project @johanbrandhorst, @awalterschulze

[achille-roussel]

Hello @awalterschulze (and other gogo protobuf maintainers), thanks for opening up this discussion!

Would you be able to give a bit more details on how would this new ownership would be ideally structured?

• Are you trying to find a new home to step down as the leaders of this project?
• Are you looking to share the responsibility with a new group of maintainers?
• Would the current maintainers be interested in a sponsorship to spend some focus time on gogo protobuf?

[awalterschulze]

Thank you. These are great questions.

Are you trying to find a new home to step down as the leaders of this project?

Yes and ideally a new home, where the employer(s) or team(s) acknowledges and gives those employee(s) at least some time to work on the project.

Are you looking to share the responsibility with a new group of maintainers?

No. All our circumstances have changed so much that this won't be possible.

Would the current maintainers be interested in a sponsorship to spend some focus time on gogo protobuf?

I know this is a No for me and Johan and highly probably a no for Donald and Jacques.

[ydnar]

We recently did some work to port a project from GoGo Protobuf to the v2 API. Aside from gogoproto.nullable (which we use quite a bit, in particular with WKTs), the blockers we identified were custom names, struct tags, and native well-known types.

Requests for these features are tracked here on the Go Protobuf repo:

• go_name: https://github.com/golang/protobuf/issues/555
• Struct tags: https://github.com/golang/protobuf/issues/52
• Standard Go types for well-known types: https://github.com/golang/protobuf/issues/414

The v2 reflection API might make implementing some of these easier than forking the project. If these features are important to you, I’d recommend weighing in on these issues.

[xen0n]

Reporter of #686 here. At work we're heavily reliant on gogo/protobuf for ergonomic gRPC APIs, and rewriting every enum and re-inventing a struct tag injector is not an option.

I tried to do some of the porting at that time, but it seems much of the machinery that the original maintainers used to track upstream is not public stuff (say, committed in this repo). On top of that it's generally hard to justify general open-source maintenance work when the department is under-staffed. We're watching this situation though, and I have just enough experience in compilers to review and debug code. If anyone steps up and decides to do the grunt work, I could probably spend a few weekends to help.

[tac0turtle]

Does someone want to compile a list of issues in a comment here & then we can work on creating a milestone to work towards?

It seems there are a few people that have stated interest in keeping this repo alive. And work towards upgrading this repo.

[johanbrandhorst]

Feel free to start with the ones linked already. We don't need a milestone, I don't think?

[johanbrandhorst]

https://github.com/gogo/protobuf/issues/695 Is another one

[tac0turtle]

where is the best place to ask questions in regards to updating? is there a chat somewhere I can join or is here the best....?

[johanbrandhorst]

The #protobuf channel on Gophers slack is probably a good place to start.

[alexshtin]

I see many teams moving out of gogo. I considered the same for our project and run simple benchmark test to compared v1, v2, and gogo: https://github.com/alexshtin/proto-bench.

The results show that gogo is still ~2 times faster and v2 is actually slower than v1. We decided to stay on gogo. Unfortunately, we don't have resources to provide much help now but we might have them later.

[aaronc]

I would love to see this project maintained and would offer to contribute as I can. We currently are maintaining a fork https://github.com/regen-network/protobuf for use in https://github.com/cosmos/cosmos-sdk. I have a few open PR's that we are using actively on our fork: #657, #658, and #426

A few of my colleagues working on Cosmos/Tendermint. @alexanderbez and @marbar3778 have chimed in here as well so we may have some limited bandwidth, but we likely wouldn't want to be the only org responsible for maintenance and upgrading to v2. If anyone else is interested let us know.

It seems from the comments that there are parallel efforts to upgrade the v2 implementation to support some of the gogo proto features, yet that implementation is slower. Either way, it would be great to see the community align on a path forward rather than splintering - either upgrading gogo to v2 or porting gogo features to v2.

[awalterschulze]

• Struct tags: golang/protobuf#52

@ydnar after years of wishfully waiting for struct tags, it seems now that golang/protobuf has decided not to implement this feature :(

[ydnar]

• Struct tags: golang/protobuf#52

@ydnar after years of wishfully waiting for struct tags, it seems now that golang/protobuf has decided not to implement this feature :(

Agreed, I’m disappointed. We kind of stirred the pot with golang/protobuf#1142, but I’m glad that the Go protobuf maintainers have at least clarified their position.

We’ve prototyped an internal tool that wraps protoc-gen-go, intercepting the CodeGeneratorRequest and CodeGeneratorResponse, parses the Go source using the ast and types packages, and modifies the output before serialization.

I’ll share what we have when we have something working for (at least) our use cases.

[tandr]

@ydnar -- I think https://github.com/srikrsna/protoc-gen-gotag does exactly what you have described

[ydnar]

I’ll share what we have when we have something working for (at least) our use cases.

Progress update: we now have a working protoc-gen-go-patch tool that does what I described above. It uses the v2 protogen package and go/ast / go/types for most of the heavy lifting, and doesn’t require forking protoc-gen-go. I hope to extract it as a separate public package sometime next week.

Currently supported options: go_tags, go_name, go_message_name, go_enum_name, and go_value_name. TODOs include gRPC, oneof support, and a patch/gogo package for gogoproto.* options that alias to the above.

More interesting features like GoGo’s WKTs like time.Time are more complicated, but it’s a fun start.

[awalterschulze]

@ydnar this sounds very promising. The TODO list as you describe should be highest priority. I hope that customtype is not necessary, but do you think casttype would be possible?

gogoproto.* options that alias to the above.

This sounds like an interesting migration story.

[awalterschulze]

@ydnar could you give an example of how users specify the tags, names, etc?

Is it also using the protobuf extensions, like https://github.com/srikrsna/protoc-gen-gotag and gogoprotobuf or do you have a new method?

[ydnar]

@ydnar could you give an example of how users specify the tags, names, etc?

Is it also using the protobuf extensions, like https://github.com/srikrsna/protoc-gen-gotag and gogoprotobuf or do you have a new method?

Exactly. It uses extensions without a package name to allow proto files to specify short option names. Here’s an example:

https://gist.github.com/ydnar/fbb49f92412ecd6bb895b36e90c9a707

[ydnar]

I hope that customtype is not necessary, but do you think casttype would be possible?

Good question. Modifying a basic type in an AST should be straightforward. More than that will require surgery. The nullable option will require changes: golang/protobuf#1142

Edit: @awalterschulze do you have any data on how often and how casttype is used, and if users specify a fully-qualified package + type selector, or just a type?

[ydnar]

Progress:

• [x] Support oneof fields.
• [x] Rename enum values when enum is renamed.
• [x] Rename oneof fields and synthesized types/methods when parent message is renamed.
• [x] Support renamed identifiers from imported packages.
• [x] Rename identifiers in doc comments in Go AST.
• [x] Support gRPC.
• [x] Add optional patch/gogo package to support GoGo Protobuf options like gogoproto.custom_name as an alias for go_name.

[moul]

Edit: @awalterschulze do you have any data on how often and how casttype is used, and if users specify a fully-qualified package + type selector, or just a type?

I used to use it this way -> https://github.com/moul/depviz/blob/cc631458b3262ff3e121fdd15b49ff42c57777bc/api/dvmodel.proto#L41

Result -> https://github.com/moul/depviz/blob/cc631458b3262ff3e121fdd15b49ff42c57777bc/internal/dvmodel/dvmodel.pb.go#L222

[awalterschulze]

@moul

I don't have data, but I expect it would mostly be qualified as in your example.

[ydnar]

I pushed an early build of protoc-gen-go-patch here: https://github.com/alta/protopatch

YMMV, but it works for us. TODO: tests, GoGo shim, more features.

[ydnar]

Update: https://github.com/alta/protopatch has a new—arguably less simple, but easier to extend—API:

• go.message.options — message options, which modify the generated Go struct for a message.
• go.field.options (also aliased to go.options) — message field options, which modify Go struct fields and getter methods.
• go.oneof.options — oneof field options, which modify struct fields, interface types, and wrapper types.
• go.enum.options — enum options, which modify Go enum types and values.
• go.value.options — enum value options, which modify Go const values.

Custom Names

import "patch/go.proto";
import "patch/go/message.proto";
import "patch/go/field.proto";
import "patch/go/enum.proto";
import "patch/go/value.proto";

message OldName {
	option (go.message.options) = {name: 'NewName'};
	int id = 1 [(go.field.options) = {name: 'ID'}];
}

enum Errors {
	option (go.enum.options) = {name: 'ProtocolErrors'};
	INVALID = 1 [(go.value.options) = {name: 'ErrInvalid'}];
	NOT_FOUND = 2 [(go.value.options) = {name: 'ErrNotFound'}];
	TOO_FUN = 3 [(go.value.options) = {name: 'ErrTooFun'}];
}

Struct Tags

message ToDo {
	int32 id = 1 (go.field.options) = {name: 'ID', tags: '`xml:"id,attr"`'}]
	string description = 2 (go.field.options) = {tags: '`xml:"desc"`'}]
}

• Tests!
• Shimming it for equivalent GoGo extensions should be straightforward.
• Adding more complex features, such as re-typing fields or casttype is more complex.

[moul]

@ydnar I gave a try to your https://github.com/alta/protopatch repo, it works very well and I like the syntax, the usage etc

fyi, for now, the only thing that prevents me to switch real projects is the lack of (gogoproto.stdtime) = true, but the project is very promising :+1:

[ydnar]

I have an initial spike on a GoGo compatibility shim, with support for one extension (gogoproto.customname). Having gone through this exercise, I’m not sure the juice is worth the squeeze. A full GoGo shim has the potential to be larger than the rest of the package. Is it worth it to maintain, when a single find & replace could potentially suffice? I’m not sure.

https://github.com/alta/protopatch/pull/8

I’m curious—which GoGo extensions do folks value most? Which could you live without? Which are obsoleted or mitigated by APIv2?