containers icon indicating copy to clipboard operation
containers copied to clipboard
verified publisher icon bitnami

kafka_jaas.conf generated withou 'Client' section

Open empikls opened this issue 3 years ago • 10 comments

Name and Version

bitnami/kafka:2.8.0

What steps will reproduce the bug?

My kafka job:

config {
        image        = "bitnami/kafka:2.8.0"
        force_pull   = true
        network_mode = "host"
      }

      env {
        BLACK_SPARKLE_AUTH_SERVICE                                              = "https://app.${meta.environment}.site/api/v1/users/auth"
        KAFKA_CFG_BROKER_ID                                                     = meta.broker_id
        KAFKA_CFG_ADVERTISED_LISTENERS                                          = "CLIENT://${NOMAD_ADDR_public},INTERNAL://${NOMAD_ADDR_private}"
        KAFKA_CFG_LISTENERS                                                     = "CLIENT://${NOMAD_ADDR_public},INTERNAL://${NOMAD_ADDR_private}"
        KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP                                = "CLIENT:SASL_SSL,INTERNAL:PLAINTEXT"
        KAFKA_CFG_INTER_BROKER_LISTENER_NAME                                    = "INTERNAL"
        KAFKA_CFG_DEFAULT_REPLICATION_FACTOR                                    = "1"
        KAFKA_CFG_LOG_RETENTION_HOURS                                           = "1"
        KAFKA_CFG_DELETE_TOPIC_ENABLE                                           = "true"
        KAFKA_CFG_TLS_CLIENT_AUTH                                               = "requested"
        KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD                                       = ""
        KAFKA_CFG_SASL_ENABLED_MECHANISMS                                       = "PLAIN"
        KAFKA_CFG_LISTENER_NAME_CLIENT_PLAIN_SASL_SERVER_CALLBACK_HANDLER_CLASS = "global.kafka.auth.AuthHandler"
        KAFKA_CERTIFICATE_PASSWORD                                              = ""
        ALLOW_PLAINTEXT_LISTENER                                                = "yes"
        KAFKA_MOUNTED_CONF_DIR                                                  = "/local"
        KAFKA_JMX_OPTS                                                          = "-javaagent:/opt/jmx-exporter.jar=${NOMAD_ADDR_exporter}:/opt/jmx.config.yml -Djava.security.auth.login.config=/local/broker_jaas.conf -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${NOMAD_IP_exporter} -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.host=localhost"
        JMX_PORT                                                                = 9999
        KAFKA_ZOOKEEPER_PROTOCOL                                                = "PLAIN"
        KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL                          = "PLAIN"
        KAFKA_ZOOKEEPER_USER                                                    = ""
        KAFKA_ZOOKEEPER_PASSWORD                                                = ""
        KAFKA_INTER_BROKER_USER                                                 = ""
        KAFKA_INTER_BROKER_PASSWORD                                             = ""
      }

It's a small part of my kafka job managed by nomad. Some parts are very similar to docker-compose.

When I run this job with KAFKA_ZOOKEEPER_PROTOCOL=PLAIN i got an error:

[2022-05-25 13:43:06,434] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/opt/bitnami/kafka/config/kafka_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2022-05-25 13:43:06,435] INFO Opening socket connection to server ip/ip:2181 (org.apache.zookeeper.ClientCnxn)
[2022-05-25 13:43:06,435] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2022-05-25 13:43:09,800] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)

Print '/opt/bitnami/kafka/config/kafka_jaas.conf' :

KafkaClient {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="user"
   password="bitnami";
   };
KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   user_user="bitnami";
   org.apache.kafka.common.security.scram.ScramLoginModule required;
   };

When I used KAFKA_ZOOKEEPER_PROTOCOL="SASL" my '/opt/bitnami/kafka/config/kafka_jaas.conf' file look like:

   KafkaClient {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="user"
   password="bitnami";
   };
KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   user_user="bitnami";
   org.apache.kafka.common.security.scram.ScramLoginModule required;
   };
Client {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="some_user"
   password="some_password";
   };

but I'm still get an error like this:

[2022-05-25 14:01:10,870] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2022-05-25 14:01:10,870] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2022-05-25 14:01:10,872] INFO Opening socket connection to server ip/ip:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2022-05-25 14:01:13,973] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-05-25 14:01:16,756] WARN Client session timed out, have not heard from server in 6003ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2022-05-25 14:01:16,863] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2022-05-25 14:01:16,863] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2022-05-25 14:01:16,864] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-05-25 14:01:16,866] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
	at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:271)
	at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:267)
	at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)
	at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)
	at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)
	at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
	at kafka.Kafka$.main(Kafka.scala:109)
	at kafka.Kafka.main(Kafka.scala)
[2022-05-25 14:01:16,868] INFO shutting down (kafka.server.KafkaServer)

I guess you forgot to add 'Client' section when KAFKA_ZOOKEEPER_PROTOCOL="PLAIN" is used.

What is the expected behavior?

No response

What do you see instead?

I guess you forgot to add 'Client' section when KAFKA_ZOOKEEPER_PROTOCOL="PLAIN" is used.

Additional information

No response

empikls avatar May 25 '22 14:05 empikls

Hi, Could you try with PLAINTEXT ? it is the default value set here

rafariossaa avatar May 26 '22 09:05 rafariossaa

@rafariossaa yes,sure. I have already tried to set PLAINTEXT to the KAFKA_ZOOKEEPER_PROTOCOL variable. The result is the same:

2022-05-26 10:06:39,910] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/opt/bitnami/kafka/config/kafka_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2022-05-26 10:06:39,910] INFO Opening socket connection to server 172.21.3.24/172.21.3.24:2181 (org.apache.zookeeper.ClientCnxn)
[2022-05-26 10:06:39,910] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2022-05-26 10:06:43,784] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-05-26 10:06:44,774] WARN Client session timed out, have not heard from server in 6002ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2022-05-26 10:06:44,880] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)

empikls avatar May 26 '22 10:05 empikls

Hi, I are you using PLAIN or PLAINTEXT ? in the jobs description pasted in the issue description you were using PLAIN . Could you check this?

rafariossaa avatar May 30 '22 10:05 rafariossaa

Hi, I'm using PLAINTEXT , sorry for confuse.

empikls avatar Jun 02 '22 13:06 empikls

Hi, Could you take a look to the configuration that is being generated in the chart ? I think there are mixed things here. The Client section is only being generated when SASL is used.

rafariossaa avatar Jun 03 '22 10:06 rafariossaa

Yes, Client section must be generated only when used SASL, but PLAINTEXT mode asked for this section too.

empikls avatar Jun 07 '22 16:06 empikls

Hi, Sorry for the delay, Could you point me to some documentation where this is indicated ? I would like to check what is being generated. Have you tried to add that section and checked if the error disappear ?

rafariossaa avatar Jun 14 '22 08:06 rafariossaa

Hi again, I used code in repo for investigation https://github.com/bitnami/bitnami-docker-kafka/blob/master/2.8/debian-10/rootfs/opt/bitnami/scripts/libkafka.sh and it can also be seen in the configuration that is formed during deployment. I've tried to add the Client section, but it didn't work, the error still appears.

empikls avatar Jun 15 '22 07:06 empikls

Hi, Thanks. I am creating an internal task in order to look deeper into this issue. We will come back as soon as we have news.

rafariossaa avatar Jun 15 '22 09:06 rafariossaa

We are going to transfer this issue to bitnami/containers

In order to unify the approaches followed in Bitnami containers and Bitnami charts, we are moving some issues in bitnami/bitnami-docker-<container> repositories to bitnami/containers.

Please follow bitnami/containers to keep you updated about the latest bitnami images.

More information here: https://blog.bitnami.com/2022/07/new-source-of-truth-bitnami-containers.html

carrodher avatar Jul 28 '22 13:07 carrodher

Hi there,

I'm working on this task, and I think the root cause may have been a misunderstanding of Kafka Security protocols.

Kafka provides the following security protocols:

  • PLAINTEXT: Un-authenticated, non-encrypted channel
  • SASL_PLAINTEXT: SASL authenticated, non-encrypted channel
  • SASL_SSL: SASL authenticated, SSL channel
  • SSL: SSL channel

Ref: https://kafka.apache.org/25/javadoc/org/apache/kafka/common/security/auth/SecurityProtocol.html

The Client section inside the kafka_jaas.conf is only required when Zookeeper is SASL authenticated, that is why the container only includes it when the variable KAFKA_ZOOKEEPER_PROTOCOL contains the string 'SASL' and user/password for Zookeeper is provided:

        if [[ "${KAFKA_ZOOKEEPER_PROTOCOL}" =~ SASL ]] && [[ -n "$KAFKA_ZOOKEEPER_USER" ]] && [[ -n "$KAFKA_ZOOKEEPER_PASSWORD" ]]; then
            cat >>"${KAFKA_CONF_DIR}/kafka_jaas.conf" <<EOF
Client {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="${KAFKA_ZOOKEEPER_USER:-}"
   password="${KAFKA_ZOOKEEPER_PASSWORD:-}";
   };
EOF
        fi

I first tried to reproduce the issue using the following docker-compose, the Client was indeed missing, but Kafka was able to start successfully as Zookeeper is configured without authentication:

version: "2"

services:
  zookeeper:
    image: bitnami/zookeeper:latest
    ports:
      - 2181:2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka: 
    image: bitnami/kafka:latest
    ports:
      - '9096:9096'
    environment:
      - KAFKA_BROKER_ID=0
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ZOOKEEPER_PROTOCOL=PLAINTEXT
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:SASL_PLAINTEXT,INTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,INTERNAL://:9096
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,INTERNAL://localhost:9096
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
    depends_on:
      - zookeeper

If I instead used the following docker-compose, with Zookeeper authentication enabled, I get the error reported in this issue:

version: "2"

services:
  zookeeper:
    image: bitnami/zookeeper:latest
    ports:
      - 2181:2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=no
      - ZOO_ENABLE_AUTH=yes
      - ZOO_SERVER_USERS=admin,bob
      - ZOO_SERVER_PASSWORDS=adminsecret,bobsecret
      - ZOO_CLIENT_USER=bob
      - ZOO_CLIENT_PASSWORD=bobsecret
  kafka: 
    image: bitnami/kafka:latest
    ports:
      - '9096:9096'
    environment:
      - BITNAMI_DEBUG=yes
      - KAFKA_BROKER_ID=0
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ZOOKEEPER_PROTOCOL=PLAINTEXT
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:SASL_PLAINTEXT,INTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,INTERNAL://:9096
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,INTERNAL://localhost:9096
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
    depends_on:
      - zookeeper

Kafka logs:

kafka_1      | [2022-10-25 09:33:29,255] WARN SASL configuration failed. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
kafka_1      | javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/opt/bitnami/kafka/config/kafka_jaas.conf'.
kafka_1      |  at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:189)
kafka_1      |  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
kafka_1      |  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
kafka_1      | [2022-10-25 09:33:29,265] ERROR [ZooKeeperClient ConfigCommand] Auth failed, initialized=false connectionState=CONNECTING (kafka.zookeeper.ZooKeeperClient)

To solve this issue, you need to configure Kafka with KAFKA_ZOOKEEPER_PROTOCOL=SASL_PLAINTEXT instead of PLAINTEXT, and provide the Zookeeper credentials accordingly, as this error means authentication failed because Zookeeper is protected with SASL.

migruiz4 avatar Oct 25 '22 09:10 migruiz4

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Nov 10 '22 01:11 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

github-actions[bot] avatar Nov 16 '22 01:11 github-actions[bot]