kafkactl icon indicating copy to clipboard operation
kafkactl copied to clipboard

Published 20 hours ago • verified publisher icon jbvmio

client/metadata got error from broker -1 while fetching metadata

Open Hubbitus opened this issue 4 years ago • 11 comments

Hello.

I am newbie and try to use kafkactl:

podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/etc/kafkactl/config.yml \
    docker.io/deviceinsight/kafkactl:v1.11.1 \
        kafkactl --verbose get topics
[kafkactl] 2020/09/22 09:02:18 Using config file: /etc/kafkactl/config.yml
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[sarama  ] 2020/09/22 09:02:18 Initializing new client
[sarama  ] 2020/09/22 09:02:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:02:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44214->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:02:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:02:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:02:48 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama  ] 2020/09/22 09:02:48 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:18 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44222->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:18 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:18 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:18 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama  ] 2020/09/22 09:03:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44228->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:48 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama  ] 2020/09/22 09:03:49 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:49 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:04:19 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44244->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:04:19 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:04:19 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:04:19 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:04:19 Closing Client
failed to create cluster admin: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Config is very simple (sed to show just non-comments and non-empty lines):

$ sed 's/#.*$//;/^$/d' .kafkactl.yaml
contexts:
  sandbox:
    name: sandbox
    brokers:
      - ecsc00a060af.epam.com:9092

current-context: sandbox

And according to wireshark dump TCP session kafkactl send metadata request and can connect to the server, but newer receive response: image

Where I am wrong?

Hubbitus avatar Sep 22 '20 09:09 Hubbitus

@Hubbitus - you're using the wrong kafkactl. In your output, deviceinsight/kafkactl, if you want to use this implementation, please try jbvmio/kafkactl.

jbvmio avatar Sep 23 '20 02:09 jbvmio

I'm very sorry for the mistake. And thank you for the response.

Meantime it changes almost nothing. Your image requires config in single place versus search in some common, but then failed with the same error without details:

$ podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/root/.kafkactl.yaml \
        docker.io/jbvmio/kafkactl:v1.0.25 \
                --verbose get topics
INFO[2020-09-23T09:03:21Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:21Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:21Z] Initializing new client                      
INFO[2020-09-23T09:03:21Z] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:22Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:27Z] client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:54082->10.6.194.60:9092: i/o timeout 
INFO[2020-09-23T09:03:27Z] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:27Z] client/metadata no available broker to send metadata request to 
INFO[2020-09-23T09:03:27Z] client/brokers resurrecting 1 dead seed brokers 
INFO[2020-09-23T09:03:27Z] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2020-09-23T09:03:27Z] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:27Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:32Z] client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:54088->10.6.194.60:9092: i/o timeout 
INFO[2020-09-23T09:03:32Z] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:32Z] client/metadata no available broker to send metadata request to 
INFO[2020-09-23T09:03:32Z] client/brokers resurrecting 1 dead seed brokers 
INFO[2020-09-23T09:03:32Z] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Hubbitus avatar Sep 23 '20 09:09 Hubbitus

And it looks like kafkactl request and receive brokers metadata, but stops on that: image Please compare with successful session list topic from kafkacat: image

Hubbitus avatar Sep 23 '20 09:09 Hubbitus

What version of Kafka are your brokers running @Hubbitus ? Can you try adding the --version 1.0.0 flag? Curious have you tried running the direct binary instead of Docker?

jbvmio avatar Sep 24 '20 12:09 jbvmio

I'm unsure about the remote broker's version (waiting answer). Use binary of kafkactl compiled locally changes nothin as I see. --version 1.0.0 also does not helps.

Still, first metadata comes and then tons of TCP retransmissions only: image

Hubbitus avatar Sep 24 '20 18:09 Hubbitus

i have the same issue

⇒  kafkactl -B 15.247.193.149:32764 --verbose
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Initializing new client
INFO[2020-10-05T19:01:53+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:53+03:00] Closed connection to broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:53+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:53+03:00] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[2020-10-05T19:01:54+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:54+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:54+03:00] Closed connection to broker 185.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:54+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:54+03:00] Closing Client
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

we have a kafka at k8s cluster and exposed via ingress and in documentation i could not find how setup sasl/ssl

zotona avatar Oct 05 '20 16:10 zotona

Sorry, maybe you may at least extend logging what happened and what wrong?

Hubbitus avatar Nov 22 '20 18:11 Hubbitus

i have the same issue

⇒  kafkactl -B 15.247.193.149:32764 --verbose
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Initializing new client
INFO[2020-10-05T19:01:53+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:53+03:00] Closed connection to broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:53+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:53+03:00] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[2020-10-05T19:01:54+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:54+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:54+03:00] Closed connection to broker 185.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:54+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:54+03:00] Closing Client
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

we have a kafka at k8s cluster and exposed via ingress and in documentation i could not find how setup sasl/ssl

@zotona You can configure sasl/ssl using the following example in your config file:

  devel-cluster:
    name: devel-cluster
    brokers:
    - devel01.example.com:9092
    - devel02.example.com:9092
    - devel03.example.com:9092
    zookeeper:
    - devel01.example.com:2181
    - devel02.example.com:2181
    - devel03.example.com:2181
    clientVersion: 2.1.1
    ssl:
      insecure: true
      tlscert: /path/to/cert.pem
      tlsca: /path/to/ca.pem
      tlskey: /path/to/key.pem
    sasl:
      username: USER
      password: PASS

jbvmio avatar Nov 27 '20 16:11 jbvmio

@Hubbitus

It seems there is either a connectivity issue or more likely the version of Kafka your brokers are running are not compatible. Did you ever confirm the version of Kafka your brokers are running?

jbvmio avatar Nov 27 '20 16:11 jbvmio

@Hubbitus - As a test, try the forcing the following client versions:

--version 0.10.1.0
--version 0.10.2.0
--version 0.11.0.0

jbvmio avatar Nov 27 '20 18:11 jbvmio 

$ kafkactl --verbose get topics                                                                                                                   
INFO[2021-03-26T17:31:36+03:00] Failed to connect to broker ecsc00a060af.epam.com:9092: dial tcp: i/o timeout 
Error connecting to Kafka: No Connectivity to Provided Brokers
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.10.1.0 get topics
INFO[2021-03-26T17:31:50+03:00] Initializing new client                      
INFO[2021-03-26T17:31:50+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:51+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:31:56+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42192->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:31:56+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:56+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:31:56+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:31:56+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:31:56+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:56+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:01+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42242->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:01+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:01+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:01+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:01+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.10.2.0 get topics
INFO[2021-03-26T17:32:15+03:00] Initializing new client                      
INFO[2021-03-26T17:32:15+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:15+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:20+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42404->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:20+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:20+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:20+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:20+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:20+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:20+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:25+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42446->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:25+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:25+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:25+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:25+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.11.0.0 get topics
INFO[2021-03-26T17:32:35+03:00] Initializing new client                      
INFO[2021-03-26T17:32:35+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:35+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:40+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42570->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:40+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:40+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:40+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:40+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:41+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:41+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:46+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42612->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:46+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:46+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:46+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:46+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 1.0.0 get topics
INFO[2021-03-26T17:32:51+03:00] Initializing new client                      
INFO[2021-03-26T17:32:51+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:51+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:56+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42694->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:56+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:56+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:56+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:56+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:56+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:56+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:33:01+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42736->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:33:01+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:33:01+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:33:01+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:33:01+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

On server: Confluent 5.3.2, Kafka 2.3.x

And off course such cluster reachable from kafkacat and confluent utilities.

Hubbitus avatar Mar 26 '21 14:03 Hubbitus