SSL Kafka handshake failed over docker

[laurafbec]

Hi everyone, Based on the example https://github.com/confluentinc/cp-docker-images/tree/5.3.3-post/examples/kafka-cluster-ssl I've tried to develop a docker-compose file with zookeeper, a broker and the connect API with SSL enabled. The certificates have been generated by using the script included in the example and the content of the docker-compose file would be the next one:

`version: '3.6'

services: zookeeper: image: confluentinc/cp-zookeeper:7.0.1 hostname: zookeeper container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ZOOKEEPER_INIT_LIMIT: 5 ZOOKEEPER_SYNC_LIMIT: 2

broker: image: confluentinc/cp-kafka:7.0.1 hostname: broker container_name: broker depends_on: - zookeeper ports: - "39093:39093" - "9093:9093" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_ADVERTISED_LISTENERS: SSL://broker:39093,SSL_HOST://localhost:9093 KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker.keystore.jks KAFKA_SSL_KEYSTORE_CREDENTIALS: broker_keystore_creds KAFKA_SSL_KEY_CREDENTIALS: broker_sslkey_creds KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker.truststore.jks KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker_truststore_creds KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: " " KAFKA_SSL_CLIENT_AUTH: required KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SSL:SSL,SSL_HOST:SSL volumes: - $PWD/secrets2:/etc/kafka/secrets

connect: image: confluentinc/cp-kafka-connect:7.0.1 hostname: connect container_name: connect depends_on: - zookeeper - broker ports: - "8083:8083" environment: CONNECT_BOOTSTRAP_SERVERS: 'broker:39093' CONNECT_REST_ADVERTISED_HOST_NAME: connect CONNECT_REST_PORT: 8083 CONNECT_GROUP_ID: compose-connect-group CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1 CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000 CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1 CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1 CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter" CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter" CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO" CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR,com.mongodb.kafka=DEBUG" CONNECT_PLUGIN_PATH: /usr/share/confluent-hub-components CONNECT_ZOOKEEPER_CONNECT: 'zookeeper:2181' CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-6.2.2.jar CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor" CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor" CONNECT_SSL_KEYSTORE_FILENAME: kafka.connect.keystore.jks CONNECT_SSL_KEYSTORE_CREDENTIALS: connect_keystore_creds CONNECT_SSL_KEY_CREDENTIALS: connect_sslkey_creds CONNECT_SSL_TRUSTSTORE_FILENAME: kafka.connect.truststore.jks CONNECT_SSL_TRUSTSTORE_CREDENTIALS: connect_truststore_creds

 volumes:
   - $PWD/mongodb-kafka-connect:/usr/share/confluent-hub-components/kafka-connect-mongodb
   - $PWD/kafka/scripts:/scripts
 command:
   - bash
   - -c
   - |
     echo "Launching Kafka Connect worker"
     /etc/confluent/docker/run &
     #
     echo "Waiting for Kafka Connect to start listening on $$CONNECT_REST_ADVERTISED_HOST_NAME"
     while [ $$(curl -s -o /dev/null -w %{http_code} http://$$CONNECT_REST_ADVERTISED_HOST_NAME:$$CONNECT_REST_PORT/connectors) -ne 200 ]; do
       echo -e $$(date) "Kafka Connect listener HTTP state: "$$(curl -s -o /dev/null -w %{http_code} http://$$CONNECT_REST_ADVERTISED_HOST_NAME:$$CONNECT_REST_PORT/connectors)" (waiting for 200)"
       sleep 5 
     done
     nc -vz $$CONNECT_REST_ADVERTISED_HOST_NAME $$CONNECT_REST_PORT
     echo -e "\n--\n+> Creating Kafka Connect MongoDB sink"
     chmod 755 /scripts/sink-connect.sh
     echo -e "Permisos cambiados"
     /scripts/sink-connect.sh 
     sleep infinity

`

When running it I get the error from broker

[2022-01-10 11:08:03,163] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /172.19.0.4 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Can anyone help me? thanks in advance

[OneCricketeer]

The error is suggesting you didn't setup SSL for Zookeeper. And your environment variables for Zookeeper seem to confirm that

The examples have moved, by the way https://github.com/confluentinc/kafka-images/blob/master/examples/kafka-cluster-ssl/docker-compose.yml

[laurafbec]

Thanks @OneCricketeer !! I've actually solved it after posting the error. I had a mismatch between kafka listeners and the certs, and now I've TLSv1.3 enabled between kafka client and broker. I haven't found examples about enabling TLS for Zokeeper of Kafka connect. Do you know where can I find some of them? Thanks again!

[OneCricketeer]

haven't found examples about enabling TLS for Zookeeper or Kafka connect.

Is this page what you're looking for?

https://docs.confluent.io/platform/current/security/security_tutorial.html

Otherwise, Zookeeper has its own official documentation, and Connect is configured like any other broker client. Both utilize KAFKA_JAVA_OPTS environment variables for setting JAAS or other JVM System properties

Connect doesn't depend on Zookeeper

[laurafbec]

Thanks again @OneCricketeer!! I actually was looking for docker-compose examples with SSL enabled between zookeeper and Kafka. On that page is described SASL for Zookeeper, but, I think that SSL is not. But thanks anyway!! I'll check everything.