SSL handshake failed with `ssl.ca.pem` option

[pravingadakh]

Description

I've been unable to connect to secured kafka broker using ssl.ca.pem option whereas ssl.ca.location works just fine. I am new to go as well as kafka, so pardon me if I am missing something obvious.

How to reproduce

So the below code with ssl.ca.location works fine. I am able to create a producer and send messages.

caFilePath := "/path/to/valid/CA/file"
producerConfigMap["ssl.ca.location"] = caFilePath

However whenever I try to to ssl.ca.pem option I am seeing ssl handshake error.

caFilePath := "/path/to/valid/CA/file"
ca, err := ioutil.ReadFile(caFilePath)
producerConfigMap["ssl.ca.pem"] = string(ca)

Error:

%3|1658172172.397|FAIL|rdkafka#producer-1| [thrd:ssl://<redacted>.]: ssl://<redacted>:9093/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 263ms in state SSL_HANDSHAKE)

Checklist

Please provide the following information:

• [ ] confluent-kafka-go and librdkafka version (LibraryVersion()): v1.8.2
• [ ] Apache Kafka broker version:
• [ ] Client configuration: ConfigMap{...}
• [ ] Operating system:
• [ ] Provide client logs (with "debug": ".." as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[ukclivecox]

I can also verify this is an issue with a set of certificates rather than a single certificate defined in ssl.ca.pem

[xuanxuandog]

I got the same issue, any one could help?

[AndrewArace]

Seeing the same behavior. SslCaPem is set to a multi chain CA:

-----BEGIN CERTIFICATE-----
(something)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(another)
-----END CERTIFICATE-----

SslCertificatePem and SslKeyPem are set as well.

I can validate the chain of SslCertificatePem + SslCaPem in command line and https://tools.keycdn.com/certificate-chain as valid (three certs, one after another) but using the programmatically set properties results in a failure:

%7|1667239774.629|BROADCAST|rdkafka#consumer-1| [thrd:ssl://<redacted>/bootstrap]: Broadcasting state change
[18:09:34 ERR] 
 Consumer Error: ssl://<redacted>/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1921: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 24ms in state SSL_HANDSHAKE)
%7|1667239775.582|CONNECT|rdkafka#consumer-1| [thrd:main]: ssl://<redacted>/bootstrap: Selected for cluster connection: coordinator query (broker has 1 connection attempt(s))
%7|1667239775.582|CGRPQUERY|rdkafka#consumer-1| [thrd:main]: Group "<redacted>": no broker available for coordinator query: intervaled in state query-coord

dotnet core 6, linux pod on openshift

[edenhill]

Try with openssl directly to get some more information, something like: openssl s_client -verify -host .. -port .. -CAfile ..

[AndrewArace]

openssl s_client -connect (redacted_host):(port) -cert cert.txt -key key.pem -CAfile chain.txt -verify_return_error

appears to connect successfully without errors, and can reconnect and renegotiate with the -reconnect switch

the "chain.txt" file has two CA root certs in it, in a valid chain. (the same string values that are read at runtime)

[edenhill]

Right, it seems to be a problem with multiple certs in the same memory blob. https://gist.github.com/dchest/574388?permalink_comment_id=1207384#gistcomment-1207384

We'll fix this in the upcoming maintenance release.

[edenhill]

We have the same issue with ssl.certificate.pem, do we need support for multiple certs for that one though?

[AndrewArace]

I don't believe there's a situation where the certificate for the main ssl certificate could ever be a chain of multiple certs.

[edenhill]

Fixed in librdkafka