pykafka icon indicating copy to clipboard operation
pykafka copied to clipboard

data of my topic in kafka is so big,when i get a consumer,i cost so much time and memoery

Open SuperLight-007 opened this issue 6 years ago • 1 comments

How can i get the last message in topic ,whe topic have 180000 messages per seconds

SuperLight-007 avatar Dec 10 '19 03:12 SuperLight-007

D:\software\IDE\python\python27\python.exe D:/workplace/health/app/platform/itoa-components/docker/itoa-gpkafka/resources/gpkafka_listener/job/kafka2GP/test.py Traceback (most recent call last): File "D:/workplace/health/app/platform/itoa-components/docker/itoa-gpkafka/resources/gpkafka_listener/job/kafka2GP/test.py", line 22, in consumer = topic.get_simple_consumer(auto_offset_reset=OffsetType.LATEST,reset_offset_on_start=True) File "D:\software\IDE\python\python27\lib\site-packages\pykafka\topic.py", line 265, in get_simple_consumer **kwargs) File "D:\software\IDE\python\python27\lib\site-packages\pykafka\simpleconsumer.py", line 244, in init self.start() File "D:\software\IDE\python\python27\lib\site-packages\pykafka\simpleconsumer.py", line 296, in start self.reset_offsets() File "D:\software\IDE\python\python27\lib\site-packages\pykafka\simpleconsumer.py", line 728, in reset_offsets response = broker.request_offset_limits(reqs) File "D:\software\IDE\python\python27\lib\site-packages\pykafka\broker.py", line 44, in wrapped raise SocketDisconnectedError pykafka.exceptions.SocketDisconnectedError

SuperLight-007 avatar Dec 10 '19 07:12 SuperLight-007