[bitnami/kafka] Kafka Containers split across multiple VMs is showing error connecting to node

[shashank-google]

Name and Version

bitnami/kafka:3.6.1

What architecture are you using?

None

What steps will reproduce the bug?

I created 3 separate VMs with docker compose. Kafka starts up on all of them. However when I try to publish a message on the topic using kafka/bin/kafka-console-producer.sh it shows me warning as below. Same is also show when I try to read or describe topic from same or different docker containers. Firewall port is open between the VMs on port 9092 and 9093.

I have no name!@7beba7d8fb71:/$ /opt/bitnami/kafka/bin/kafka-console-producer.sh --topic mytopic --bootstrap-server localhost:9092
>[2024-02-27 04:18:44,959] WARN [Producer clientId=console-producer] Error connecting to node 596abf026523:9092 (id: 2 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: 596abf026523: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:122)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1030)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:64)
	at org.apache.kafka.clients.producer.internals.Sender.awaitNodeReady(Sender.java:562)
	at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:483)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:331)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:246)
	at java.base/java.lang.Thread.run(Thread.java:840)

Where 596abf026523 is container id which is deployed on another VM.

Setup as below:

VM Hostname: my-vm-0

version: "3"
name: kafka-cluster
services:
  kafka:
    image: 'docker.io/bitnami/kafka:latest'
    ports:
      - '9092:9092'
      - '9093:9093'
    environment:
      # KRaft settings
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@my-vm-0:9093,1@my-vm-1:9093,2@my-vm-2:9093
      - KAFKA_KRAFT_CLUSTER_ID=abcdefghijklmnopqrstuv
      # Listeners
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      # Clustering
      - KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=3
      - KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
      - KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=2
    volumes:
      - /opt/bitnami-kafka-data:/bitnami/kafka

VM Hostname: my-vm-1 While rest of config is same, only changed value is as below

    environment:
      # KRaft settings
      - KAFKA_CFG_NODE_ID=1

VM Hostname: my-vm-2 While rest of config is same, only changed value is as below

    environment:
      # KRaft settings
      - KAFKA_CFG_NODE_ID=2

SSH in one of the containers and create the topic.

What is the expected behavior?

No warnings or errors. Kafka Controller should not be looking up other containers based on container-id.

What do you see instead?

I see errors as below

I have no name!@7beba7d8fb71:/$ /opt/bitnami/kafka/bin/kafka-console-producer.sh --topic mytopic --bootstrap-server localhost:9092
>[2024-02-27 04:18:44,959] WARN [Producer clientId=console-producer] Error connecting to node 596abf026523:9092 (id: 2 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: 596abf026523: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:122)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1030)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:64)
	at org.apache.kafka.clients.producer.internals.Sender.awaitNodeReady(Sender.java:562)
	at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:483)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:331)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:246)
	at java.base/java.lang.Thread.run(Thread.java:840)

Additional information

No response

[shashank-google]

@carrodher Request you to please share your thoughts

[shashank-google]

@CeliaGMqrz bumping this up

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[CeliaGMqrz]

Hi @shashank-google

Thanks for your patience.

Sorry for the delay in my response. To help me better understand the issue you're facing, could you please let me know which Kubernetes platform you are using? This will allow me to reproduce your environment. Based on the information provided, it seems like a dns error. Have you tried using the hostname directly, as suggested in the docs?

Thanks for your feedback!

[shashank-google]

HI @CeliaGMqrz We are not able to use kubernetes platform (due to various reasons). Therefore we have installed 2 separate VMs with docker. Then deploying the two containers separately. Also due to other constraints on the environment we cannot bridge the docker networks on two VMs.

In addition, also note that we further found that even the client (which is separately deployed on another VM) was also not able to interact kafka-single-docker for the same reason.

[juan131]

Hi @shashank-google

• Are you simultaneously starting the Kafka containers on the 3 VMs?
• Is there connectivity between these VMs?
• How did you configure the DNS? I mean, where did you configure those hostnames for the VMs (my-vm-0, my-vm-1 and my-vm-2? It looks like the containers aren't able to resolve these hostnames and, as a consequence, they can't reach the other Kafka brokers.

[shashank-google]

Hi @juan131 I have since then moved to apache's official kafka 3.7 container which works fine for me. The bitnami kafka containers assume container-id as the hostname and since client does not know anything about container-id it is not getting resolved. Overall bitnami container dod not worked for me and I believe I have provided all the detailed information to reproduce the same.

[juan131]

Hi @shashank-google

I'm glad you were able to solve the issue using a different image.

Please note there's no limitation to use the container-id as the hostname using Bitnami images though. They'll try to resolve the provided hostname to an IP consulting the available DNS servers. In the examples we use that approach sin Docker networking automatically generates DNS records associating the hostnames in this way, however that's not a limitation in the container image.