Backend example not working with mojaloop - kafka init containers not finishing

[mihai-tarabuta-yt]

Hi there,

Followed the instructions here https://docs.mojaloop.io/technical/deployment-guide/ to deploy mojaloop (v15.0.0) on Azure AKS. It does not work out of the box and the problem appears to be the kafka init containers not being able to connect to kafka controllers. kafka init container logs: Waiting for Kafka... [2024-04-29 23:25:54,518] WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) Exception in thread "main" java.lang.RuntimeException: Request METADATA failed on brokers List(kafka:9092 (id: -1 rack: null)) at kafka.admin.BrokerApiVersionsCommand$AdminClient.sendAnyNode(BrokerApiVersionsCommand.scala:159) at kafka.admin.BrokerApiVersionsCommand$AdminClient.findAllBrokers(BrokerApiVersionsCommand.scala:182) at kafka.admin.BrokerApiVersionsCommand$AdminClient.awaitBrokers(BrokerApiVersionsCommand.scala:174) at kafka.admin.BrokerApiVersionsCommand$.execute(BrokerApiVersionsCommand.scala:61) at kafka.admin.BrokerApiVersionsCommand$.main(BrokerApiVersionsCommand.scala:55) at kafka.admin.BrokerApiVersionsCommand.main(BrokerApiVersionsCommand.scala)

Kafka Controller logs [2024-04-29 23:37:29,574] INFO [SocketServer listenerType=BROKER, nodeId=0] Failed authentication with /10.244.0.185 (channelId=10.244.0.227:9092-10.244.0.185:45134-41935) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) ... [2024-04-29 23:37:49,862] INFO [SocketServer listenerType=BROKER, nodeId=0] Failed authentication with /10.244.0.185 (channelId=10.244.0.227:9092-10.244.0.185:51610-41937) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) When this happens the containers "wait-for-kafka" will chew up all the CPU resources and may give the wrong impression that the cluster/node is saturated. tried with a different version of contaner init (solsson/kafka:3.3.0 instead of solsson/kafka:2.8.1) but the same problem exists.