SASL/SCRAM-SHA-256 authentication failure

[ben-marengo-msmg]

Name and Version

bitnami/kafka:3.2

What steps will reproduce the bug?

docker-compose.yaml

version: '3'
services:

  zookeeper:
    container_name: zookeeper
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    environment:
      - ZOO_ENABLE_AUTH=yes
      - ZOO_SERVER_USERS=user
      - ZOO_SERVER_PASSWORDS=password
      - ZOO_CLIENT_USER=user
      - ZOO_CLIENT_PASSWORD=password

  kafka:
    container_name: kafka
    image: docker.io/bitnami/kafka:3.2
    ports:
      - "9093:9093"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=false

      - KAFKA_ZOOKEEPER_PROTOCOL=SASL
      - KAFKA_ZOOKEEPER_USER=user
      - KAFKA_ZOOKEEPER_PASSWORD=password

      - KAFKA_CLIENT_USERS=user
      - KAFKA_CLIENT_PASSWORDS=password

      - KAFKA_INTER_BROKER_USER=user
      - KAFKA_INTER_BROKER_PASSWORD=password

      - KAFKA_CFG_LISTENERS=INTERNAL://:9092,EXTERNAL://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,EXTERNAL://localhost:9093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL

      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-256
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-256
      - KAFKA_CFG_SECURITY_INTER_BROKER_PROTOCOL=SASL_PLAINTEXT

    volumes:
      - bitnami-config:/opt/bitnami/kafka/config

    depends_on:
      - zookeeper

  kafka_init:
    container_name: kafka_init
    image: docker.io/bitnami/kafka:3.2
    command:
      - /opt/bitnami/kafka/bin/kafka-topics.sh
      - --create
      - --bootstrap-server
      - kafka:9092
      - --topic
      - my-topic
      - --partitions
      - "1"
      - --replication-factor
      - "1"
    depends_on:
      - kafka
    environment:
      - KAFKA_OPTS=-Djava.security.auth.login.config=/opt/bitnami/kafka/config/kafka_jaas.conf

    volumes:
      - bitnami-config:/opt/bitnami/kafka/config:ro

volumes:
  bitnami-config:

What is the expected behavior?

topic gets created

What do you see instead?

kafka_init container fails to create topic kafka container log shows lots of these:

[2022-07-21 10:26:27,096] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1002] Failed authentication with /172.25.0.4 (channelId=172.25.0.3:9092-172.25.0.4:46096-58) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

Additional information

$ cat /opt/bitnami/kafka/config/kafka_jaas.conf
KafkaServer {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="user"
   password="password";
   };
Client {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="user"
   password="password";
   };

[OneCricketeer]

I think you can ignore the Zookeeper related connection logs, unless the broker is crashing because it cannot connect to Zookeeper...

It's worth pointing out that depends_on: kafka will only wait for the container to start, not the broker process within it to start listening, so you init container is probably having a race condition.

Could you try to use docker-compose exec kafka bash -c "kafka-topics.sh --create ..." instead after the container is started?

Also KafkaServer JAAS config needs to be on the broker. You seem to be missing that.

[ben-marengo-msmg]

hey @OneCricketeer - thanks for your help!

It's worth pointing out that depends_on: kafka will only wait for the container to start, not the broker process within it to start listening, so you init container is probably having a race condition.

noted, thanks. i actually couldnt find a good healthcheck to verify the broker was ready. could you suggest one?

Could you try to use docker-compose exec kafka bash -c "kafka-topics.sh --create ..." instead after the container is started?

i tried this and get the same result; lots of the following from the kafka container

[2022-07-26 10:03:16,684] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1001] Failed authentication with /172.21.0.3 (channelId=172.21.0.3:9092-172.21.0.3:45990-29) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

Also KafkaServer JAAS config needs to be on the broker. You seem to be missing that.

im not sure what you mean? i have a KafkaServer section in /opt/bitnami/kafka/config/kafka_jaas.conf, shown in the 'Additional information' section of my question. this file seems to be generated when the broker boots by me supplying env vars: KAFKA_CLIENT_USERS,KAFKA_CLIENT_PASSWORDS,KAFKA_INTER_BROKER_USER,KAFKA_INTER_BROKER_PASSWORD

[OneCricketeer]

this file seems to be generated when the broker boots

Ah, thought you were cat-ing a file outside of the container. Nevermind.

couldnt find a good healthcheck to verify the broker was ready

kafka-topics --list is what I've seen most Docker HEALTHCHECK values set, with a reasonable timeout, but still, I am not sure that depends_on will wait for the healthcheck either.

I personally haven't tried setting up Zookeeper with auth enabled. My comments were mostly related to the init container potentially not being able to start in the correct order, and it may also need a --command-config file option with more properties to fully authenticate to the broker.

[carrodher]

We are going to transfer this issue to bitnami/containers

In order to unify the approaches followed in Bitnami containers and Bitnami charts, we are moving some issues in bitnami/bitnami-docker-<container> repositories to bitnami/containers.

Please follow bitnami/containers to keep you updated about the latest bitnami images.

More information here: https://blog.bitnami.com/2022/07/new-source-of-truth-bitnami-containers.html

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[ben-marengo-msmg]

hey @carrodher / @alemorcuq - just checking this is still on your radar after it got closed as stale

[alemorcuq]

Sorry, @ben-marengo-msmg, we lost track of this. I've found this StackOverflow thread mentioning a config parameter to be missing. Can you check that?

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.