Support the newest message formats for Kafka versions that support them

[dantswain]

trafficstars

This will be a fairly large undertaking.

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-TheAPIs

Each message type has one or more versions, and the versions were implemented in a particular version of the broker. The correlation is very inconsistent (e.g., v0 always starts with the first version of the broker that implements a message, so for Kafka 0.11 we may have v0 of one message and v3 of some other message).

So far we are "lucky" in that the message formats are all backwards compatible, but there are operational consequences of this. See #244 for an example.

[jbruggem]

This is now possible, since we can ask the server for the message version it supports. See an example here: https://github.com/kafkaex/kafka_ex/pull/320/files#diff-5ed86f44a0ff14f109f61199ad39cfc1L107

[dantswain]

So after brainstorming a little with @joshuawscott, I decided to write some code to pull schema definitions from https://github.com/klarna/kafka_protocol (the underlying library used by brod) and use macros to generate Elixir-idiomatic structs and serde code: https://github.com/dantswain/kayrock

I'm still tweaking it, but I believe it should work. I have ideas about how it could be integrated with KafkaEx, but I'm super busy lately so if anyone would like to take that on feel free to reach out to me. Ideally this would mean we can basically get rid of the entire KafkaEx.Protocol stack and not have to manually integrate new api versions (aside from any API logic we want to implement on this side). It's limited to whatever messages/versions kafka_protocol has implemented, which lags a little bit but not as much as KafkaEx has ;)

[joshuawscott]

@dantswain do you think this can be closed now?