kafka-python
kafka-python copied to clipboard
Issues with Amazon MSK
Hello,
I am currently using version 2.0.2
to publish messages to an Amazon MSK broker version 2.3.1
but I am running into a wall
when I get this exception (when calling the send
method)
File \"/usr/local/lib/python3.8/site-packages/kafka/producer/kafka.py\", line 702, in _wait_on_metadata\n raise Errors.KafkaTimeoutError(\nkafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs."
My code works perfectly when I run it locally using the confluentinc/cp-kafka:5.4.3
docker image as a Kafka broker and I am wondering if there are some compatibility issues with this MSK version mentioned above. I notice that 2.3 isn't in the compatibility matrix.
I have verified that the topic, which I am publishing to, exists on the broker as this was the first thing suggested when I googled this issue. Also verified that the container running the producer can connect to the MSK cluster (by telnetting to the 9092 port)
ignore me, figured out what the problem was. the container was reading the topic name from an env var which was mangled and I totally oversaw this.