org.apache.kafka.clients.NetworkClient Error connecting to node broker:29092 (id: -1 rack: null) java.net.UnknownHostException: broker: Name or service not known

[dash7ou]

control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available. rest-proxy | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed rest-proxy | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501070911, tries=1, nextAllowedTryMs=1606501071012) timed out at 1606501070912 after 1 attempt(s) rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available. ksqldb-server | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed ksqldb-server | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501071531, tries=1, nextAllowedTryMs=1606501071632) timed out at 1606501071532 after 1 attempt(s) ksqldb-server | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. schema-registry | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed schema-registry | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501072795, tries=1, nextAllowedTryMs=1606501072897) timed out at 1606501072797 after 1 attempt(s) schema-registry | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null) control-center | java.net.UnknownHostException: broker: Name or service not known control-center | at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) control-center | at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) control-center | at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515) control-center | at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958) control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224) control-center | at java.base/java.lang.Thread.run(Thread.java:834) control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null) control-center | java.net.UnknownHostException: broker control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958) control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224) control-center | at java.base/java.lang.Thread.run(Thread.java:834) control-center | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed control-center | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501079587, tries=1, nextAllowedTryMs=1606501079698) timed out at 1606501079598 after 1 attempt(s) control-center | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null) control-center | java.net.UnknownHostException: broker control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958) control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281) control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224) control-center | at java.base/java.lang.Thread.run(Thread.java:834) rest-proxy | [main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list. rest-proxy | java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s) rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) rest-proxy | at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149) rest-proxy | at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150) rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s) rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null) control-center | java.net.UnknownHostException: broker control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)

[dash7ou]

@ybyzek I have an issue when running docker-compose file for cp-all-in-one folder, on ec2 amazon, it runs fine for me locally.

[cosbor11]

This happens for me too

[konfortes]

Same here for local run. osx 10.15.5 Docker 3.0.3(51017)

[wscardua]

me too

osx 10.15.7 docker 3.0.4 (51218)

[ybyzek]

Things to check:

1. Is the broker up? (docker-compose ps)
2. Can the other container ping the broker container? (docker-compose exec control-center ping broker)
3. Are there any errors in the logs (search through docker-compose logs)
4. Which branch of this repo is checked out? Make sure to run latest (latest or 6.0.1-post)

[mhbrk]

docker-compose logs -f broker

Attaching to broker broker | ===> User broker | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) broker | ===> Configuring ... broker | ===> Running preflight checks ... broker | ===> Check if /var/lib/kafka/data is writable ... broker | ===> Check if Zookeeper is healthy ... broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=broker broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=11.0.9.1 broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc. broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu11-ca broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/share/java/cp-base-new/paranamer-2.8.jar:/usr/share/java/cp-base-new/netty-resolver-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-library-2.13.2.jar:/usr/share/java/cp-base-new/metrics-core-2.2.0.jar:/usr/share/java/cp-base-new/audience-annotations-0.5.0.jar:/usr/share/java/cp-base-new/kafka-clients-6.0.1-ccs.jar:/usr/share/java/cp-base-new/commons-cli-1.4.jar:/usr/share/java/cp-base-new/slf4j-api-1.7.30.jar:/usr/share/java/cp-base-new/jopt-simple-5.0.4.jar:/usr/share/java/cp-base-new/jackson-databind-2.10.5.jar:/usr/share/java/cp-base-new/netty-transport-4.1.48.Final.jar:/usr/share/java/cp-base-new/jackson-module-scala_2.13-2.10.5.jar:/usr/share/java/cp-base-new/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-logging_2.13-3.9.2.jar:/usr/share/java/cp-base-new/jackson-dataformat-yaml-2.10.5.jar:/usr/share/java/cp-base-new/argparse4j-0.7.0.jar:/usr/share/java/cp-base-new/gson-2.8.6.jar:/usr/share/java/cp-base-new/scala-collection-compat_2.13-2.1.6.jar:/usr/share/java/cp-base-new/jackson-datatype-jdk8-2.10.5.jar:/usr/share/java/cp-base-new/jackson-annotations-2.10.5.jar:/usr/share/java/cp-base-new/utility-belt-6.0.1.jar:/usr/share/java/cp-base-new/netty-buffer-4.1.48.Final.jar:/usr/share/java/cp-base-new/common-utils-6.0.1.jar:/usr/share/java/cp-base-new/scala-java8-compat_2.13-0.9.1.jar:/usr/share/java/cp-base-new/zookeeper-jute-3.5.8.jar:/usr/share/java/cp-base-new/zookeeper-3.5.8.jar:/usr/share/java/cp-base-new/kafka_2.13-6.0.1-ccs.jar:/usr/share/java/cp-base-new/zstd-jni-1.4.4-7.jar:/usr/share/java/cp-base-new/jackson-core-2.10.5.jar:/usr/share/java/cp-base-new/lz4-java-1.7.1.jar:/usr/share/java/cp-base-new/netty-handler-4.1.48.Final.jar:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar:/usr/share/java/cp-base-new/jackson-module-paranamer-2.10.5.jar:/usr/share/java/cp-base-new/snappy-java-1.1.7.3.jar:/usr/share/java/cp-base-new/netty-common-4.1.48.Final.jar:/usr/share/java/cp-base-new/jackson-dataformat-csv-2.10.5.jar:/usr/share/java/cp-base-new/snakeyaml-1.26.jar:/usr/share/java/cp-base-new/netty-transport-native-epoll-4.1.48.Final.jar:/usr/share/java/cp-base-new/netty-codec-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-reflect-2.13.2.jar broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.19.121-linuxkit broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=appuser broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/appuser broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/appuser broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=120MB broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1992MB broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=126MB broker | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@a7e666 broker | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation broker | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes broker | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled= broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.24.0.2:2181. Will not attempt to authenticate using SASL (unknown error) broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.24.0.2:2181: Connection refused broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.24.0.2:2181. Will not attempt to authenticate using SASL (unknown error) broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.24.0.3:59678, server: zookeeper/172.24.0.2:2181 broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.24.0.2:2181, sessionid = 0x1000024ccb10000, negotiated timeout = 40000 broker | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000024ccb10000 closed broker | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000024ccb10000 broker | ===> Launching ... broker | ===> Launching kafka ... broker | [2021-01-17 21:17:02,315] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) broker | [2021-01-17 21:17:03,638] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) broker | [2021-01-17 21:17:03,679] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092 broker | at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:260) broker | at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:271) broker | at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:2297) broker | at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2376) broker | at kafka.server.KafkaConfig.(KafkaConfig.scala:2354) broker | at kafka.server.KafkaConfig.(KafkaConfig.scala:1807) broker | at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34) broker | at kafka.Kafka$.main(Kafka.scala:68) broker | at kafka.Kafka.main(Kafka.scala) broker exited with code 1

[mhbrk]

also it doesn't have - "29092:29092" in this part: https://github.com/confluentinc/cp-all-in-one/blob/6.0.1-post/cp-all-in-one/docker-compose.yml#L20

[ybyzek]

@mhbrk from the error logs:

broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092

It looks like 9092 is reused twice, this is not allowed as described in the message. So it appears your local docker-compose.yml file has been modified. Please update it to match what is in GitHub (which is validated to work) and try again.

[DavoBR]

about the port 29092, i am facing with the following error

container console output:

confluentinc_cp-server_6.1.0.txt

[OneCricketeer]

also it doesn't have - "29092:29092" in this part

broker hostname isn't resolvable by the host, anyway, so a port forward is not the correct location to debug the problem

As the logs say, your listeners config cannot use the same port. Without this, your broker dies as well as all other services, such as control center, that depend on it

You may explicitly define KAFKA_LISTENERS, otherwise the default behavior in the Docker image is to use the advertised listeners, and replace the host names with 0.0.0.0

These get printed in the logs when the container starts https://github.com/confluentinc/confluent-docker-utils/blob/master/confluent/docker_utils/cub.py#L469

[tooptoop4]

@dash7ou did u solve?

[nicolepastrana]

This is what worked for me in my M1 Chip:

• zookeeper image: confluentinc/cp-zookeeper:latest.arm64
• broker image: confluentinc/cp-server:latest.arm64
• schema-registry image: confluentinc/cp-schema-registry:latest.arm64
• connect image: cnfldemos/cp-server-connect-datagen:0.5.0-6.2.0 [I haven't tested with the latest cnfldemos/cp-server-connect-datagen:0.5.3-7.1.0]
• control-center image: confluentinc/cp-enterprise-control-center:latest.arm64
• ksqldb-server image: confluentinc/cp-ksqldb-server:latest.arm64
• ksqldb-cli image: confluentinc/cp-ksqldb-cli:latest.arm64
• ksql-datagen image: confluentinc/ksqldb-examples:latest.arm64
• rest-proxy image: confluentinc/cp-kafka-rest:latest.arm64

https://github.com/confluentinc/cp-all-in-one/blob/7.1.1-post/cp-all-in-one/docker-compose.yml