kafka-rest icon indicating copy to clipboard operation
kafka-rest copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

SSL handshake fails

Open lucasmarfe opened this issue 6 years ago • 2 comments

I have a kafka cluster on docker using confluent images. I am using docker-compose to build the containers.

When I tried to run the container it starts but can't communicate with any broker due to SSL handshake failed. I don't know if I miss some configuration

kafka-rest-proxy_1  | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-1] Failed authentication with /10.98.198.103 (SSL handshake failed)
kafka-rest-proxy_1  | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error
kafka-rest-proxy_1  | org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
kafka-rest-proxy_1  | Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
kafka-rest-proxy_1  |   at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
kafka-rest-proxy_1  |   at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
kafka-rest-proxy_1  |   at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
kafka-rest-proxy_1  |   at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:448)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:313)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:170)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
kafka-rest-proxy_1  |   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
kafka-rest-proxy_1  |   at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1131)
kafka-rest-proxy_1  |   at java.lang.Thread.run(Thread.java:748)
kafka-rest-proxy_1  | Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
kafka-rest-proxy_1  |   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
kafka-rest-proxy_1  |   at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
kafka-rest-proxy_1  |   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
kafka-rest-proxy_1  |   at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
kafka-rest-proxy_1  |   at java.security.AccessController.doPrivileged(Native Method)
kafka-rest-proxy_1  |   at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:402)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:484)
kafka-rest-proxy_1  |   at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:340)
kafka-rest-proxy_1  |   ... 7 more
kafka-rest-proxy_1  | Caused by: java.security.cert.CertificateException: No subject alternative names present
kafka-rest-proxy_1  |   at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:145)
kafka-rest-proxy_1  |   at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
kafka-rest-proxy_1  |   at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
kafka-rest-proxy_1  |   at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
kafka-rest-proxy_1  |   at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
kafka-rest-proxy_1  |   at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
kafka-rest-proxy_1  |   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
kafka-rest-proxy_1  |   ... 16 more
kafka-rest-proxy_1  | [kafka-admin-client-thread | adminclient-1] ERROR org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -2 (/10.98.198.103:19092) failed authentication due to: SSL handshake failed

My Kafka brokers are configured as follows:

kafka1: image: confluentinc/cp-kafka:5.2.2 container_name: kafka1 ports:

  • "19092:19092" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: XXX:12181,XXX:12181,XXX:12181 KAFKA_ADVERTISED_LISTENERS: SSL://XXXX:19092 KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker1.keystore.jks KAFKA_SSL_KEYSTORE_CREDENTIALS: broker1_keystore_creds KAFKA_SSL_KEY_CREDENTIALS: broker1_sslkey_creds KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker1.truststore.jks KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker1_truststore_creds KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: "" KAFKA_SSL_CLIENT_AUTH: required KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL KAFKA_SECURITY_PROTOCOL: SSL volumes:
  • ./../../secrets:/etc/kafka/secrets

I am trying to bring a Confluent REST Proxy API into another container using the configurations:

kafka-rest-proxy: image: confluentinc/cp-kafka-rest:5.2.2 hostname: kafka-rest-proxy ports:

  • "18082:18082" environment: KAFKA_REST_LISTENERS: "http://0.0.0.0:18082" KAFKA_REST_ZOOKEEPER_CONNECT: XXX:12181,XXX:12181,XXX:12181 KAFKA_REST_HOST_NAME: kafka-rest-proxy KAFKA_REST_BOOTSTRAP_SERVERS: SSL://XXX:19092,SSL://XXX:19092,SSL://XXX:19092 KAFKA_REST_CLIENT_SECURITY_PROTOCOL: SSL KAFKA_REST_CLIENT_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka.broker1.keystore.jks KAFKA_REST_CLIENT_SSL_KEYSTORE_PASSWORD: XXX
    KAFKA_REST_CLIENT_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.broker1.truststore.jks KAFKA_REST_CLIENT_SSL_TRUSTSTORE_PASSWORD: XXX KAFKA_REST_CLIENT_SSL_KEY_PASSWORD: XXX KAFKA_REST_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka.producer.keystore.jks KAFKA_REST_SSL_KEYSTORE_PASSWORD: XXX KAFKA_REST_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.producer.truststore.jks KAFKA_REST_SSL_TRUSTSTORE_PASSWORD: XXX volumes:
  • ./../../secrets:/etc/kafka/secrets

lucasmarfe avatar Jun 26 '19 11:06 lucasmarfe

Are u still facing this issue ? Looks like, the certificates of kafka has the problem. kafka certificates must have the name with which it is accessed by the kafka rest added as part of the CommonName(CN) or the SAN (subject alternative names). else client(Kafka rest in this case) upon handshake will compare the domain it is trying to connect to and the aliases presented by the kafka in SSL handshake via CN or SAN and fails if a matching entry is not found.

kaushiksrinivas avatar Dec 11 '19 06:12 kaushiksrinivas

I am seeing this same error. It appears that the error is caused because the only the IP address of the kafka-rest-proxy is being transmitted and not the hostname:

[2020-08-28 03:47:00,332] INFO [SocketServer brokerId=1] Failed authentication with /192.168.112.5 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

My kafka config is:

  kafka1:
    image: confluentinc/cp-kafka:5.5.1
    hostname: kafka1
    command: "bash -c 'if [ ! -f /etc/kafka/secrets/kafka1.keystore.jks ]; then echo \"ERROR: Did not find SSL certificates in /etc/kafka/secrets/\" && exit 1 ; else /etc/confluent/docker/run ; fi'"
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_SSL://kafka1:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_SSL:SSL,LISTENER_DOCKER_EXTERNAL:SSL
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_SSL
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SSL
      KAFKA_SASL_ENABLED_MECHANISMS: SSL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_SSL_KEYSTORE_FILENAME: kafka1.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: creds
      KAFKA_SSL_KEY_CREDENTIALS: creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka1.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: creds
      KAFKA_SSL_CLIENT_AUTH: "required"
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
    volumes:
      - ./full-stack/kafka1/data:/var/lib/kafka/data
      - ./keys:/etc/kafka/secrets
    depends_on:
      - zoo1

and kafka-rest-proxy config is:

  kafka-rest-proxy:
    image: confluentinc/cp-kafka-rest:5.5.1
    hostname: kafka-rest-proxy
    ports:
      - "8082:8082"
    volumes:
      - ./keys:/etc/kafka/secrets
    environment:
      #KAFKA_REST_ZOOKEEPER_CONNECT: zoo1:2181
      KAFKA_REST_LISTENERS: http://0.0.0.0:8082/
      KAFKA_REST_SCHEMA_REGISTRY_URL: http://kafka-schema-registry:8081/
      KAFKA_REST_HOST_NAME: kafka-rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: SSL://kafka1:19093
      KAFKA_REST_CLIENT_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka-rest-proxy.truststore.jks
      KAFKA_REST_CLIENT_SSL_TRUSTSTORE_PASSWORD: testadmin
      KAFKA_REST_CLIENT_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka-rest-proxy.keystore.jks
      KAFKA_REST_CLIENT_SSL_KEYSTORE_PASSWORD: testadmin
      KAFKA_REST_CLIENT_SSL_KEY_PASSWORD: testadmin
    depends_on:
      - zoo1
      - kafka1
      - kafka-schema-registry

The only thing I can think of is there is a problem where the hostname (in this case kafka-rest-proxy) is not being transmitted when trying to connect

kbreidenbachRN avatar Aug 28 '20 03:08 kbreidenbachRN