afkak
afkak copied to clipboard
ciena
Kafka client written in Twisted Python
I'm documenting this quickly for internet reference and to gather feedback on the plan to address the items outlined here, which also align with those stated in #45. Our immediate...
`KakfaClient.close()` terminates outstanding connections and fails operations that were waiting on requests in those brokers, but the client may internally retry due to those failures. This can lead to both:...
The `Consumer` argument `fetch_max_wait_time` sets the upper bound of how long the FetchRequest polls for results. The default is 150 milliseconds, meaning that `Consumer` issues more than 6 fetch requests...
Example scripts (e.g. [consumer_example](https://github.com/ciena/afkak/blob/master/consumer_example)) do not run in Python3: ``` (venv) » ./consumer_example 172.16.1.13:9092 main function encountered error Traceback (most recent call last): File "/home/ubuntu/sandbox/afkak/venv/lib/python3.5/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks result...
These are redundant: https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/kafkacodec.py#L46 https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/common.py#L38-L39
The `Consumer` class's `fetch_max_wait_time` parameter defaults to `FETCH_MAX_WAIT_TIME = 100` milliseconds: https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/consumer.py#L46-L47 Internally, `Consumer` calls `KafkaClient.send_fetch_request()`. This method has a `max_wait_time` parameter with a default value of `KafkaClient.DEFAULT_FETCH_SERVER_WAIT_MSECS`: https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/client.py#L151-L158 It...
The `FastMurmur2` extra pulls in C implementation of the Murmur2 hash function. It should be mentioned in the readme. Mention varying platform support: the pyhash library requires PyPy 6.0 or...
https://pypi.org/project/afkak/ doesn't mention the changelog. It should.
This issue tracks changes to Afkak required to support Kafka 2.0.0. See the [upgrade guide](https://kafka.apache.org/20/documentation.html#upgrade): ## KIP-283 Message Down-Conversion requires client changes > [KIP-283](https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion) improves message down-conversion handling on Kafka...
I'd like to get some feedback on this -- the basics seem pretty straightforward and I'd like see if there's anything I missed. I need to figure out what to...
