pykafka
pykafka copied to clipboard
Parsely
Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance.
Consider a mode of operation for the consumer in which some partitions being inaccessible (possibly due to `GroupLoadInProgressCode` errors) does not cause the consumer to crash. `fetch_offsets` in particular doesn't...
Hi there, I want to create and delete topics from kafka server based on some conditions. I have written a simple script just to verify topic creation and deletion using...
Reason is : Need to limit database connection access by grouping key access together. Hence, the idea of processing message by mini-batch. msg_list = consume(10msg,...) We can do ourself as...
``` INFO:pykafka.balancedconsumer:Rebalancing consumer "pykafka-88ab9f20-80db-493c-9253-998a14cfb0d6" for topic "my-replicated-topic". INFO:pykafka.managedbalancedconsumer:Sending JoinGroupRequest for consumer id 'pykafka-88ab9f20-80db-493c-9253-998a14cfb0d6' INFO:pykafka.balancedconsumer:pykafka-88ab9f20-80db-493c-9253-998a14cfb0d6: Balancing 2 participants for 10 partitions. Owning 5 partitions. DEBUG:pykafka.balancedconsumer:My partitions: ['my-replicated-topic-0-2', 'my-replicated-topic-0-5', 'my-replicated-topic-0-8', 'my-replicated-topic-1-3', 'my-replicated-topic-1-0']...
We hit a production issue with a stuck `pykafka`-based consumer that I tracked down to a single message that was larger than the consumer's `fetch_message_max_bytes`, but smaller than the broker's...
The [`kafka_tools`](https://github.com/Parsely/pykafka/blob/b1d958bd05c1dad491e8243e032f87b10eb6d528/pykafka/cli/kafka_tools.py) module is now big and developed enough to warrant some automated testing. Add basic tests that exercise the CLI entrypoints to that module.
Tornado
I would like to consume and produce messages using PyKafka from within a Tornado application. I am potentially willing to contribute code for this. I have a few questions. 1....
As proven by libraries like psycopg2 it is possible to provide a way to integrate a C extension with gevent. This may or may not require cooperation on librdkafka's part...
**PyKafka version**: master ( commit - 9ae97cbcad4198b1b1ef58c25e4f51657c135688 ) **Kafka version**: 0.9.0 We have Kafka configured with a 3 node cluster. Also we have a producer producing messages, and a managed...
https://github.com/apache/kafka/pull/2048 This change to Kafka introduces verifiable clients, which define an interface clients can implement to work against Kafka's own integration tests. We should investigate what it will take to...