SocketDisconnectedError for creating and deleting topics from remote kafka server

[girishsv1986]

Hi there, I want to create and delete topics from kafka server based on some conditions. I have written a simple script just to verify topic creation and deletion using pykafka protocol.py CreateTopicsRequest and DeleteTopicsRequest classes

Issue I'm facing is - Script works fine with local kafka server instances. For remote kafka server script able to establish connection and get topics, but always fails with the SocketDisconnectedError error while creating topic and deleting topic

Traceback (most recent call last): File "pykafa_create_delete_topic.py", line 24, in <module> resp = client.cluster.controller_broker.create_topics([CreateTopicRequest(topic_name, 2, 3, [], [])], 5000) File "venv/lib/python2.7/site-packages/pykafka/broker.py", line 45, in wrapped return fn(self, *args, **kwargs) File "venv/lib/python2.7/site-packages/pykafka/broker.py", line 571, in create_topics return future.get(CreateTopicsResponse) File "venv/lib/python2.7/site-packages/pykafka/handlers.py", line 75, in get raise self.error SocketDisconnectedError: <broker <host_ip>:39092> attached is my script pykafa_create_delete_topic.zip

Can someone help me to figure out where it's going wrong? Also, is there a better way create and delete topics than what I have done

Thanks in Advance.

PyKafka version: 2.7.0 Kafka version: 1.1.0

[emmettbutler]

@girishsv1986 Thanks for the question. The standard way to use PyKafka to create and delete topics is via the CLI. The script you're using does look correct. Verify that the machine running the script has access to the kafka brokers, and that you're indeed using Kafka 1.1.0 on the remote cluster.

[girishsv1986]

Remote machine is running with same kafka version and similar configuration. As I mentioned earlier, I'm able to connect to remote kafka server and get list of topics from remote server. Error occurs only when I'm trying to create or delete topic.

(Script works fine with local kafka server instances) Same issue occurs when I run same script from remote machine and tries to connect to my local kafka instance(which eventually becomes remote kafka server for script again).