cp-ansible icon indicating copy to clipboard operation
cp-ansible copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

Issue with KRaft Configuration: ZooKeeper Settings Appear in server.properties

Open masoudtayefeh opened this issue 6 months ago • 1 comments

I'm encountering an issue with configuring Kafka in KRaft mode using the Ansible role for Confluent Platform. Despite specifying KRaft mode in my hosts.yml file, the generated server.properties file still includes ZooKeeper-related settings.

Debian Version: 12.6 cp-ansible: 7.7.0-post

hosts.yml:

kafka_controller:
  hosts:
    kafka-1:
    kafka-2:
    kafka-3:

kafka_broker:
  hosts:
    kafka-1:
    kafka-2:
    kafka-3:

schema_registry:
  hosts:
    kafka-miscellaneous:

kafka_rest:
  hosts:
    kafka-3:

kafka_connect:
  hosts:
    kafka-miscellaneous:

all:
  vars:
    ansible_connection: ssh
    ansible_user: kafka
    ansible_become: true
    ansible_ssh_private_key_file: /home/kafka/.ssh/id_rsa
    jmxexporter_enabled: true
    jolokia_enabled: true
    proxy_env:
       http_proxy: http://proxy:8118
       https_proxy: http://proxy:8118
    kafka_broker_custom_properties:
       num.io.threads: 15

After running the Ansible playbook with the above configuration, the server.properties file on Kafka brokers still contains ZooKeeper-related configurations.

Error that i get after running ansible:

TASK [confluent.platform.kafka_controller : Check Kafka Metadata Quorum]
fatal: [kafka-1]: FAILED! => {"changed": false, "cmd": "/usr/bin/kafka-metadata-quorum --bootstrap-server kafka-1:9091  --command-config /etc/controller/client.properties describe --replication\n", "delta": "0:01:01.149704", "end": "2024-08-19 10:20:01.372824", "msg": "non-zero return code", "rc": 1, "start": "2024-08-19 10:19:00.223120", "stderr": "[2024-08-19 10:19:01,068] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:01,172] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:01,274] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:01,576] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:02,079] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:20:00,070] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\norg.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum\njava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)\n\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)\n\tat org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.handleDescribeReplication(MetadataQuorumCommand.java:158)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:106)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:62)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:57)\nCaused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum", "stderr_lines": ["[2024-08-19 10:19:01,068] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. ((org.apache.kafka.clients.NetworkClient)", "org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum", "java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum", "\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)", "\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)", "\tat org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.handleDescribeReplication(MetadataQuorumCommand.java:158)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:106)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:62)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:57)", "Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum"], "stdout": "", "stdout_lines": []}

And here is confluent-server service logs:

Aug 05 17:28:34 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:35,693] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:35,694] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:36,795] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:36,797] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:37,898] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:37,899] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:39,000] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:39,002] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:40,103] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:40,105] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:41,206] INFO Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:41,207] WARN Session 0x0 for server localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:42,308] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:42,309] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:42,662] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,410] INFO Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,514] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,515] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,516] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,518] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:216)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:116)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2853)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2959)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:1065)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.startup(KafkaServer.scala:355)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka$.main(Kafka.scala:112)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka.main(Kafka.scala)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,519] INFO shutting down (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,537] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,538] INFO shut down completed (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,538] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:216)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:116)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2853)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2959)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:1065)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.startup(KafkaServer.scala:355)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka$.main(Kafka.scala:112)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka.main(Kafka.scala)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,539] INFO shutting down (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 systemd[1]: confluent-server.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 17:28:43 kafka-1 systemd[1]: confluent-server.service: Failed with result 'exit-code'.
Aug 05 17:28:43 kafka-1 systemd[1]: confluent-server.service: Consumed 3.602s CPU time.

masoudtayefeh avatar Aug 19 '24 07:08 masoudtayefeh