poseidon
poseidon copied to clipboard
Publisher topic generates successfully but message doesn't post.
Consumer works like a charm but the producer is another story. When attempting to post to a topic the library will generate the topic, if not already there, but the message to be posted never posts. Infact I get the following error.
/Users/foo/.rbenv/versions/2.1.0/gemsets/kafka/gems/poseidon-0.0.5/lib/poseidon/sync_producer.rb:66:in `send_messages': Failed to send all messages: [#<Poseidon::Message:0x007fbb7a9f1ef0 @struct=#<struct Poseidon::Protocol::MessageWithOffsetStruct offset=0, message=#<struct Poseidon::Protocol::MessageStruct magic_type=0, attributes=0, key=nil, value="hello">>, @topic="test">] remaining (RuntimeError)
from /Users/foo/.rbenv/versions/2.1.0/gemsets/kafka/gems/poseidon-0.0.5/lib/poseidon/producer.rb:163:in `send_messages'
from producer.rb:6:in `<main>'
My implementation is very simple.
require 'poseidon'
producer = Poseidon::Producer.new(['foo:9092'], 'test_producer')
message_buffer = []
message_buffer << Poseidon::MessageToSend.new('test_topic', 'bar')
producer.send_messages(message_buffer)
Kafka install: kafka_2.9.2-0.8.1.1
Please advise...
I have the same problem...
I tried to send message with other languages, Scala, Node.js and Java. but all of them failed to send message to Kafka on external server.
So I think our problem occurs due to server settings.
I too have the same problem :P . Please let me know if you find the cause or the solution.
I solved this. and found poseidon has no problems... ^^; If you use Kafka with docker-kafka, I can tell you the solution.
Hey @tagia0212 ,the issue resolved by itself ,once I restarted the kafka server. But, I couldn't find the reason behind the error. Though I do not use docker, but some insight on the issue might help.
@tagia0212, could you elaborate on the solution for docker-kafka? Just curious to know.
@daluu OK, but I know the solution for kafka-docker, not docker-kafka. sorry.
I cloned the repository onto the server, and run docker up -d
. When I failed to send topic from Poseidon to Kafka, I have not set config.
So, I rewrite docker-compose.yml.
environment, KAFKA_ADVERTISED_HOST_NAME must be set global IP Address of the server!
After I rerun the docker container, I could send the topics to Kafka.
Not a fix, but at least it would then be possible to capture an explicit error instead of a RuntimeError: https://github.com/bpot/poseidon/pull/101
this would be useful to merge