[protobuf] add support for Protobuf v3

[kubacech]

Hello, I want to ask in what state is adding support for protobufs v3? V3 is released for some time now and in docs is a mention, that there is plan to add support for it.

Thanks, Jakub

[cowtowncoder]

@kubacech I don't think anyone is working on protobuf v3 support at this point. I don't have time myself, but can help if anyone is interested. I am not sure how easy/possible it will be support both v2 and v3 with same package; I hope that is plausible. If not we'd need a separate format module (which would be ok too).

[wujimin]

i have a try to add basic map support currently map key can be Integer/Long/String

can i raise a PR for this?

[cowtowncoder]

@wujimin Since this is not related to question of protobuf 3 support, could you please add it either to an existing issue that is related to Map support (I think there is one), or, if none exists, file a new one?

[wujimin]

very sorry for the late reply.

after add map feature (https://github.com/huaweicse/jackson-dataformats-binary/pull/1) i had tried to add "Any" feature, but found too difficult to do this. in fact, my map feature solution is not so good too.

after a performance test, i found, compare to protoStuff, jackson solution is not so good

so i had gave up jackson solution, and will write new serializer/deserializer based on jackson and protoStuff

[cowtowncoder]

@wujimin I would be interested in performance part, as Jackson backend should have performance comparable to protostuff (when binding to POJOs). Not necessarily faster, but similar. So mostly interested in kind of test you tried since maybe I could find things to improve here.

Thank you for your update, even if you are not planning to use Jackson backend. And good luck with development using protostuff, that is a good library too.

[wujimin]

i have removed jackson test part. (maybe my new map feature cause a bad performance, :)) and my new solution just only finished most serializers(scalars/collection/map/message/any, not include enum/oneOf yet.), have not start deserializers

after all features finished, i will do more end to end performance test, include jackson and then update to this issue.

[cowtowncoder]

@wujimin Thanks!

[wujimin]

hi, i have finished most features

PR in my project: https://github.com/apache/incubator-servicecomb-java-chassis/pull/930 there are performance data in the page.

performance test code: https://github.com/apache/incubator-servicecomb-java-chassis/pull/930/commits/4ab91bcbe3f1dcfa1b56b34b7cf0b3a35a021a49

[vincentfree]

Any progress with support for V3?

[cowtowncoder]

Usually when there is progress there are notes here so I assume no progress unfortunately.

[jedvardsson]

Proto3 syntax have been available in an official Protobuf release since 2016. I can’t find any note that the syntax is not official. They’ve added back optional but other than that it has been quite stable, right? Is anyone using proto2 nowadays.

So, what would the efforts be for implementing a Jackson module for it. Any benefits of waiting for Jackson 3?

[cowtowncoder]

The only blocker here is the lack of resources -- someone to write the code, test. Support need not wait until Jackson 3; nor is there anything specific within Jackson 3 (master branch) to make it easier to support.

So if there is someone with an itch and time, it'd be great to get support added. But I do not personally have time to work on this, unfortunately. As much as I would like to see it happen. I try to find time help others, however, when they produce something.

So: The usual OSS Credo applies here -- Contributions Welcome! (new module, or update to this one if backwards compatible, either way is fine)

[jedvardsson]

I appreciate the effort you put into Jackson. I’ll see if there was a something I can do to help. I’ll need to study the existing code to understand the scope.

[cowtowncoder]

Thank you @jedvardsson (and apologies for slow follow up). Please let me know if you need any help! And any findings, suggestions might be helpful too -- perhaps there are incremental things that could be added. If I remember correctly, even with ProtoV2 there are some things (like convention to support Maps) that could use improvements.

[vogt31337]

I'd also like to know what the state of the support of v3 is? Is it possible to use jackson to read/write v3 if only v2 features were used (for example no map, only optional, enum and repeated)?

[cowtowncoder]

My understanding is that Protobuf v3 is backwards-compatible at binary encoding level, so I think it may work. But I have not tried this myself so if anyone tries it out, please add a comment here.

[vogt31337]

I'll give it a try and I'll report here.

[cowtowncoder]

Thank you @vogt31337

[itsmoonrack]

Just a message to cheer up if there is any developers wiling to take this work, i've been using protobuf jackson and looked very promising (i love the API and pluggability). But the lack of proto 3 support was a blocker for us. This is a feature that is much needed I think and seeing the comments here I guess many developers would love to see that happening in the Jackson ecosystem. So is my little 2023 up of this ticket.

Have we any high level directions ?

• what library do we want to use as underlying ?
• do we want to support full scope as a first release ? optionals and so on ? or a limited set as a beta version
• do we want to release a new proto3 binary module ? or proto2 and proto3 should be on the same module ?

[cowtowncoder]

Thank you @itsmoonrack ! One of first things might be to upgrade library used for reading proto definitions -- this because currently used implementation is deprecated and there is a replacement (but not drop-in wrt api), see #121.

As to feature set, it really depends on complexity implementation: I'm ok (and might prefer) incremental addition of supported features.