confluent-kafka-python icon indicating copy to clipboard operation
confluent-kafka-python copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

`enable.ssl.certificate.verification` must be a string and not a boolean

Open yzhan289 opened this issue 2 years ago • 4 comments

Description

The enable.ssl.certificate.verification configuration for AdminClient only takes strings "true"/"false" rather than the Python booleans True/False. We noticed that even though we would set enable.ssl.certificate.verification: False in our configuration, we would end up with the following error:

%3|1692896138.459|FAIL|dd-agent#producer-1| [thrd:sasl_ssl://192.168.36.20:9071/bootstrap]: sasl_ssl://192.168.36.20:9071/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 2ms in state SSL_HANDSHAKE)

The default value of enable.ssl.certificate.verification (according to librdkafka's configuration) is true, so maybe after the config is passed from confluent-kafka-python to librdkafka, the boolean False is converted to the default string "true"?

Based on https://github.com/confluentinc/confluent-kafka-python/issues/1346, one could assume that enable.ssl.certificate.verification should take Python booleans, although from https://github.com/confluentinc/confluent-kafka-python/issues/938 and https://github.com/confluentinc/confluent-kafka-python/issues/1494, other users might be running into this issue too.

The library should either clarify that enable.ssl.certificate.verification should be a string or update the implementation to accept a boolean.

How to reproduce

  1. Configure a kafka cluster that requires SSL but doesn't have a server cert signed by a CA (can be self signed).
  2. Create an AdminClient connection that uses SSL but set enable.ssl.certificate.verification: False (Python boolean not string).

Checklist

Please provide the following information:

  • [x] confluent-kafka-python and librdkafka version: ~2.2.0~ 2.0.2
  • [x] Apache Kafka broker version: N/A
  • [x] Client configuration: security.protocol: "sasl_ssl", sasl.mechanism: "PLAIN", enable.ssl.certificate.verification: False, sasl.username: "<username>", sasl.password: "<password>", sasl.kerberos.principal: "kafka@localhost", sasl.kerberos.service.name: "kafka"
  • [x] Operating system: N/A
  • [ ] Provide client logs (with 'debug': '..' as necessary)
  • [ ] Provide broker log excerpts
  • [ ] Critical issue

yzhan289 avatar Aug 28 '23 14:08 yzhan289

I don't get the problem that you are mentioning with the conf value. It is properly setting the value to false when I gave Python boolean False. Can you verify with setting 'debug': 'conf' in the client configuration?

My code has this Config ->

conf = {'bootstrap.servers': broker, 'group.id': group, 'auto.offset.reset': 'earliest', 'enable.auto.offset.store': False, 'debug': 'conf', 'enable.ssl.certificate.verification': False }

And debug logs are ->

2023-08-29 15:31:10,491 DEBUG INIT [rdkafka#consumer-1] [thrd:app]: librdkafka v2.2.0-5-g8bec8d-dirty (0x20200ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x40000) 2023-08-29 15:31:10,491 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: Client configuration: 2023-08-29 15:31:10,491 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: client.software.name = confluent-kafka-python 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: client.software.version = 2.2.0-rdkafka-2.2.0-5-g8bec8d-dirty 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: metadata.broker.list = localhost:9092 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: debug = conf 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: error_cb = 0x7f012ba1a940 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: log_cb = 0x7f012ba1a7d0 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: log.queue = true 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: opaque = 0x7f01299a0680 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: default_topic_conf = 0x5648f223e810 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: enable.ssl.certificate.verification = false 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: group.id = asdfgs 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: session.timeout.ms = 6000 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: enable.auto.offset.store = false 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: rebalance_cb = 0x7f012ba13d70 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: offset_commit_cb = 0x7f012ba152c0 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: Default topic configuration: 2023-08-29 15:31:10,492 DEBUG CONF [rdkafka#consumer-1] [thrd:app]: auto.offset.reset = smallest Traceback (most recent call last):

pranavrth avatar Aug 29 '23 10:08 pranavrth

Hey @pranavrth , sorry for the delay. Also, I should clarify that this is happening on confluent-kafka-python 2.0.2 (but also happens on 2.2.0):

We tested with:

    config = {
        "security.protocol": "sasl_ssl",
        "enable.ssl.certificate.verification": False,
        "sasl.mechanism": sasl_mechanism,
        "sasl.username": sasl_plain_username,
        "sasl.password": sasl_plain_password,
        "sasl.kerberos.principal": sasl_kerberos_principal,
        "sasl.kerberos.service.name": sasl_kerberos_service_name,
        "bootstrap.servers": broker,
        "socket.timeout.ms": 30000,
        "debug": "conf",
    }

This is the result:

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: Client configuration:

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: client.id = dd-agent

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: client.software.name = confluent-kafka-python

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: client.software.version = 2.0.2-rdkafka-2.0.2

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: metadata.broker.list = 192.168.0.30:9071

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: debug = conf

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: socket.timeout.ms = 30000

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: error_cb = 0x7fe8dada7740

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: background_event_cb = 0x7fe8dad9f680

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: opaque = 0x7fe8da5a9250

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: security.protocol = sasl_ssl

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: sasl.mechanisms = PLAIN

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: sasl.kerberos.service.name = kafka

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: sasl.kerberos.principal = kafka@localhost

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: sasl.username = [redacted]

%7|1693420528.190|CONF|dd-agent#producer-1| [thrd:app]: sasl.password = [redacted]

%3|1693420528.193|FAIL|dd-agent#producer-1| [thrd:sasl_ssl://192.168.0.30:9071/bootstrap]: sasl_ssl://192.168.0.30:9071/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1921:tls_process_server_certificate error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 2ms in state SSL_HANDSHAKE)

yzhan289 avatar Aug 30 '23 19:08 yzhan289

Running some tests, I found that True, "true", and "false" allow enable.ssl.certificate.verification = x to appear in the config, but False seems to fail?

yzhan289 avatar Aug 30 '23 20:08 yzhan289

I have the same issue with Kafka commands.( kafka-topics.sh )

version: 2.13-3.4.0

client.properties

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
enable.ssl.certificate.verification=false
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
   username="xxxxxxxxxxxxxx" \
   password="xxxxxxxxxxxxxx";

Log

Error while executing topic command : SSL handshake failed
[2024-04-30 10:23:35,998] ERROR org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
        ... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 24 more

vector-mj avatar Apr 30 '24 10:04 vector-mj