protobuf

protobuf copied to clipboard

Is your feature request related to a problem? Please describe. The default generator optimizes for code size, which is unsuitable for some use cases.

Describe the solution you'd like I'd like the first-party optimize_for file option to be supported. As per https://developers.google.com/protocol-buffers/docs/proto3#options it would include implementing different codegen based on the 3 options:

SPEED CODE_SIZE LITE_RUNTIME (might not apply at all to Go files)

According to the documentation, SPEED is the default, but changing that would potentially be considered a backwards-incompatible change, so I'm happy to leave CODE_SIZE the default in that case.

Describe alternatives you've considered Alternatives include using the gogo/protobuf project to generate message specific messages. This is widely used in industry where speed is desired over code size, and one of the major reasons for its popularity.

Additional context This could be another opportunity to merge parts of the gogo/protobuf codebase into this repo, and potentially win back a lot of users of gogo/protobuf to the official implementation.

I'd say that the real feature request is "make protos faster"; supporting optimize_for might be one part of such performance improvements, but is not a goal in and of itself.

Improving proto performance is something that we are actively working on. (Protos consume a lot of CPU in Google, so there's a pretty direct line between performance improvements and money saved.) We've made some pretty significant improvements over the past year with the table-driven (un)marshal implementation, and should hopefully have more to come in the not-too-distant future.

@neild thanks for your input, but my understanding behind the choice of reflection-based marshalling instead of per-message generated methods was because of the smaller generated code footprint. It also seems clear to me that the easiest way to get reliable speedups in the marshalling is to generate message-specific marshalling logic (proof: see gogo/protobuf benchmarks).

With that in mind, I think there is a clear argument for letting the user choose the obvious compromise of larger generated files for faster marshalling.

One problem with the optimize_for file option is that it assumes that the owner of a given .proto file is the best team to decide how the generated code should be optimized. However, it turns out that this is not always true. As a general observation, the owner of a .proto file often believes that speed is of high importance for their own .proto files (since they are the primary user), but tends to view speed as less important for some of their transitive dependencies (since they are not the primary use-case). However, this may contradict the decision of their transitive dependencies, who independently may believe that their own .proto should be optimized for speed. This becomes a problem at scale if most (or all) files are marked as being optimized for speed, leading to significant binary bloat.

I don't have any concrete answers, but it seems that the users of a .proto are in a better position to decide how it should be optimized. This could perhaps be informed through profiling on a per-binary use-case.

That's a great argument against the option, but not the user having a choice. Could it be a generator argument?