Add support for new message format

[xstevens]

Kafka uses a new messaging format in the most recent versions. This is denoted by the magic byte being set to 1 and is followed by some message attributes which let you know if compression was used, and if so what type. See the Kafka design guide for more details.

FYI: The ruby client looks like they've updated for this already.

[sthomas-github]

This might be along the same lines. When I run a consumer against our kafka installation (0.7.2), I get:

Traceback (most recent call last): File "readkafka.py", line 10, in for message in consumer.loop(): File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 46, in loop messages = self.consume() File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 38, in consume self.send_consume_request() File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 68, in send_consume_request self.write(self.encode_request()) File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 64, in encode_request return struct.pack('>HH%dsiQi' % length, self.request_type, length, self.topic, self.partition, self.offset, self.max_size) struct.error: integer out of range for 'Q' format code

I'd like to use python for kafka interrogation, but we will have to fall back to Java unless I can get around issues like this, so any help would be appreciated.

Thanks,

Steve