Unable to get started with example

[poida]

trafficstars

I'm following the example cluster setup instructions described here: https://github.com/travisjeffery/jocko/tree/master/examples/cluster.

The cluster seems to startup ok. There are message about leader election and no errors.

However when I try and read or write data to it with kafkacat I get IO errors.

From kafkacat terminal:

➜  jocko git:(master) ✗ tail -f /var/log/Xorg.0.log | kafkacat -b 127.0.0.1:9001 -t xorglog -z snappy
% Auto-selecting Producer mode (use -P or -C to override)

%3|1486551458.751|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9001/bootstrap]: 127.0.0.1:9001/1: Receive failed: Disconnected
%3|1486551458.762|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9002/2]: 127.0.0.1:9002/2: Receive failed: Disconnected
^C% ERROR: Local: Broker transport failure: 127.0.0.1:9001/1: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: 127.0.0.1:9002/2: Receive failed: Disconnected
%3|1486551463.762|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9003/3]: 127.0.0.1:9003/3: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: 127.0.0.1:9003/3: Receive failed: Disconnected
% ERROR: Local: All broker connections are down: 3/3 brokers are down: terminating

From one of the brokers:

2017/02/08 22:04:23 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:7948
2017-02-08 22:04:26 [DEBUG] jocko: correlation id [1], request size [30], key [3]
2017-02-08 22:04:31 [INFO] jocko: Conn read failed: read tcp 127.0.0.1:9001->127.0.0.1:55294: i/o timeout
2017-02-08 22:04:34 [DEBUG] jocko: correlation id [1], request size [30], key [3]
2017-02-08 22:04:34 [DEBUG] jocko: correlation id [2], request size [30], key [3]
2017-02-08 22:04:36 [DEBUG] jocko: correlation id [2], request size [30], key [3]
2017-02-08 22:04:36 [DEBUG] jocko: correlation id [3], request size [30], key [3]
2017-02-08 22:04:42 [DEBUG] jocko: correlation id [1], request size [30], key [3]
2017/02/08 22:04:47 [DEBUG] memberlist: TCP connection from=127.0.0.1:47232
2017-02-08 22:04:47 [INFO] jocko: Conn read failed: read tcp 127.0.0.1:9001->127.0.0.1:55312: i/o timeout
2017/02/08 22:04:47 [DEBUG] memberlist: TCP connection from=127.0.0.1:47238
2017-02-08 22:04:52 [INFO] jocko: Conn read failed: read tcp 127.0.0.1:9001->127.0.0.1:55320: i/o timeout
2017/02/08 22:04:53 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:7948

Any ideas? What's an easy way I can test this?

[travisjeffery]

Hm, looks like something's up with the metadata. Can you send me all the commands I need to run to reproduce?

[poida]

@travisjeffery here are the commands I ran, with each in a separate terminal.

Broker 1

./jocko --debug \
          --logdir="/tmp/jocko1" \
          --tcpaddr=127.0.0.1:9001 \
          --raftdir="/tmp/jock1/raft" \
          --raftaddr=127.0.0.1 \
          --raftport=8001 \
          --serfport=7946 \
          --id=1

Broker 2

./jocko --debug \
          --logdir="/tmp/jocko2" \
          --tcpaddr=127.0.0.1:9002 \
          --raftdir="/tmp/jocko2/raft" \
          --raftaddr=127.0.0.1 \
          --raftport=8002 \
          --serfport=7947 \
          --serfmembers=127.0.0.1:7946 \
          --id=2

Broker 3

./jocko --debug \
          --logdir="/tmp/jocko3" \
          --tcpaddr=127.0.0.1:9003 \
          --raftdir="/tmp/jocko3/raft" \
          --raftaddr=127.0.0.1 \
          --raftport=8003 \
          --serfport=7948 \
          --serfmembers=127.0.0.1:7946 \
          --id=3

Kafka cat producer:

tail -f /var/log/Xorg.0.log | kafkacat -b 127.0.0.1:9001 -t xorglog

Kafkacat consumer:

kafkacat -b 127.0.0.1:9001 -t xorglog

[jrnt30]

I think you need to create the topic ahead of time for it to work properly.

[poida]

Thx for adding the createtopic command, but still no luck, similar errors to above. This is using latest master, with updated example commands.

Producer Errors:

createtopic git:(master) ✗ tail -f /var/log/Xorg.0.log | kafkacat -b 127.0.0.1:9001 -t xorglog -P -p 0
%3|1487670194.355|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9001/bootstrap]: 127.0.0.1:9001/1: Receive failed: Disconnected
%3|1487670194.367|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9201/3]: 127.0.0.1:9201/3: Receive failed: Disconnected
%3|1487670199.368|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9101/2]: 127.0.0.1:9101/2: Receive failed: Disconnected
%3|1487670199.455|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9001/bootstrap]: 127.0.0.1:9001/1: Receive failed: Disconnected
%3|1487670204.456|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9201/3]: 127.0.0.1:9201/3: Receive failed: Disconnected
%3|1487670204.468|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9101/2]: 127.0.0.1:9101/2: Receive failed: Disconnected
%3|1487670209.469|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9001/bootstrap]: 127.0.0.1:9001/1: Receive failed: Disconnected
%3|1487670209.556|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9201/3]: 127.0.0.1:9201/3: Receive failed: Disconnected
%3|1487670214.557|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9101/2]: 127.0.0.1:9101/2: Receive failed: Disconnected

Consumer Errors:

  createtopic git:(master) ✗ kafkacat -b 127.0.0.1:9001 -t xorglog -C -p 0
%3|1487670201.891|FAIL|rdkafka#consumer-1| [thrd:127.0.0.1:9001/bootstrap]: 127.0.0.1:9001/1: Receive failed: Disconnected
%3|1487670201.902|FAIL|rdkafka#consumer-1| [thrd:127.0.0.1:9201/3]: 127.0.0.1:9201/3: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: 127.0.0.1:9001/1: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: 127.0.0.1:9201/3: Receive failed: Disconnected
%3|1487670206.902|FAIL|rdkafka#consumer-1| [thrd:127.0.0.1:9101/2]: 127.0.0.1:9101/2: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: 127.0.0.1:9101/2: Receive failed: Disconnected
% ERROR: Local: All broker connections are down: 3/3 brokers are down: terminating

[matthew-d-jones]

I tried producing and consuming simple messages with two python libraries kafka-python and confluent-kafka and see the same behaviour as poida. (using latest master)

[gertcuykens]

kafkacat -C -b 127.0.0.1:9092 -t test_topic

%3|1509902329.177|PROTOERR|rdkafka#consumer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/1: Remaining data after message set parse: 164 bytes
^C%3|1509902329.278|PROTOERR|rdkafka#consumer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/1: Protocol parse failure at 4/168 (rd_kafka_fetch_reply_handle:2749) (incorrect broker.version.fallback?)

[sparques]

I see this is still open--I'm seeing similar to this problem. I can open a separate issue if it's entirely unrelated.

jocko: node id: 0: 2018/12/17 11:51:59 ERROR: error when flushing the buffer: write udp 127.0.0.1:53893->127.0.0.1:6831: write: connection refused

Looking at the output of netstat -uape it doesn't seem like I have anything that's actually listening on udp port 6831.