18F
Investigate impact of protocol buffers on users
Investigate the impact on users to implement protocol buffers vs. versioning.
Acceptance Criteria: some documentation attached to this story of what the impact on users would be if we tried protocol buffers vs. versions. Also outline the benefits of this approach vs. the cost.
Might be good to loop in @annalee, @ertzeid, or @ccostino on this one as well
@mbland, did you use Protocol Buffers at all while at Google, and if so, what were your impressions? Good, bad, indifferent? In reading the link you had posted (https://developers.google.com/protocol-buffers/docs/overview?hl=en) it looks interesting to me but I am curious how they panned out in practice.
@ccostino Yes, we used them everywhere. @ertzeid can talk about them too. They were great; we not only used them for RPCs, but for data storage and as a configuration language--much the same way folks use JSON everywhere today, but in a much more time- and space-efficient manner that also provided a robust mechanism to avoid versioning hell.
There's a text-based format that should be easy to use for things like .about.yml (except we'd have to rename it .about.proto, probably). They're type-safe, which would eliminate some of the .about.yml fragility, and the fact that individual fields are "tagged" with integer IDs enables programs expecting an older version to safely ignore fields from a newer version, and programs expecting a newer version may choose to check for or ignore fields from an older version.
One downside is that building programs that use protos require a new tool in the toolchain, a protocol compiler. In practice, it's not that onerous, but here's the killer, perhaps: Compilers are currently only available for C++, Java, Python, and C#. However, there is a protobuf-json project which may help.
There are options for Ruby and Node, as well. Such as:
https://github.com/ruby-protobuf/protobuf https://www.npmjs.com/package/node-protobuf
Ah, cool! I was only checking the official Google distribution and did a quick search for "protocol buffers json". Interesting. This may change the equation a bit, especially since I expect we can browserify the Node package. (And if I could get @hzeller to open up gqui or other protobuf tools, that might present interesting opportunities for data-heavy projects...but I digress. ;-)
So there's a hint of what the text format for protobufs looks like, but I can't find more extensive documentation. Suffice it to say, it resembles JSON quite a bit. Lemme see if I can get an example .about.proto together and parse it with that node package.
Thinking on the versioning issue some more, I'm kind of on the fence as to whether versioning will actually be needed after we hammer out some of these early issues. I'd expect most changes to be additive, not so much breaking, and YAML seems fine for that use case. Protobufs are wonderful for many, many reasons, but YAML might remain just fine for our purposes here. (Though, the type safety enforced by protobuf compilers may still be worth consideration, in light of the need for validation.)
That said, I'm still open-minded towards the possibility, if we really do find versioning to be a compelling solution to a real problem. Note the discussion in 18F/api-standards#5.