AWS MSK broker certificate validation fails with custom CA

[jnt2007]

Steps to reproduce:

1. Deploy Amazon MSK instance 3.8.1 version.
2. Configure SASL_SSL security protocol.
3. Check the broker's certificate.
4. Download CA certificate that issued the broker's certificate.
5. Use this CA certificate in the consumer's SASL session configuration as ssl.ca.location.

Actual result: SSL handshake failed: ssl/statem/statem_clnt.c:2103:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 297ms in state SSL_HANDSHAKE) (_SSL)

Expected result: SSL handshake successful, connection established

$ openssl verify -CAfile ~/Downloads/Amazon-RSA-2048-M03.pem /tmp/server.crt
/tmp/server.crt: OK

P.S. It might be a problem with librdkafka, but I have no idea how to reproduce it.

full_debug.log Log

2025-04-04T16:41:10.877332236Z %3|1743784870.877|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 232ms in state SSL_HANDSHAKE)
2025-04-04T16:41:11.710793051Z %3|1743784871.710|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 366ms in state SSL_HANDSHAKE)
2025-04-04T16:41:12.533172399Z %3|1743784872.532|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 184ms in state SSL_HANDSHAKE)
2025-04-04T16:41:13.501485909Z %3|1743784873.500|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 273ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
2025-04-04T16:41:14.664694469Z %3|1743784874.664|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 323ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
2025-04-04T16:41:17.632499779Z %3|1743784877.632|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 294ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)

Dockerfile to reproduce

FROM python:3.13-alpine

# Install wget (to download the certificate) and pip dependencies
RUN apk add --no-cache --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community \
    wget gcc libc-dev librdkafka-dev=2.8.0-r0 \
    && pip install confluent-kafka==2.8.2

# Download the Amazon RSA 2048 M03 certificate
RUN mkdir /certs && wget -O /certs/amazon-rsa2048-m03.pem \
    "https://www.amazontrust.com/repository/Amazon-RSA-2048-M03.pem"

# Copy the consumer script into the container
COPY consumer.py /app/consumer.py
WORKDIR /app

CMD ["python", "consumer.py"]

consumer.py

from confluent_kafka import Consumer

config = {
    'bootstrap.servers': "b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096,"
                         "b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096,"
                         "b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096",
    'security.protocol': "SASL_SSL",
    'sasl.mechanism': "SCRAM-SHA-512",
    'sasl.username': "user",
    'sasl.password': "REDACTED",
    'ssl.ca.location': "/certs/amazon-rsa2048-m03.pem",
    'group.id': "my-consumer-group",
    'auto.offset.reset': 'earliest',
    'debug': 'all',
}

consumer = Consumer(config)
consumer.subscribe(["my_topic"])  # Replace with your topic name

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print("Consumer error: {}".format(msg.error()))
            continue
        print('Received message: {}'.format(msg.value().decode('utf-8')))
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

[and-ratajski]

Same here! Actually we get a lot of those in a pretty random times - I can see correlation with high load on brokers.

@ConfluentInteractiveDev - could you please check it?

%4|1744713756.100|FAIL|rdkafka#producer-3| [thrd:sasl_ssl://b-3.xxxxxxxxxx.amazonaws.com:90]: sasl_ssl://b-3.xxxxxxxxxxxxxx.amazonaws.com:9098/3: Connection setup timed out in state SSL_HANDSHAKE (after 39755ms in state SSL_HANDSHAKE)

3|1744784524.301|FAIL|rdkafka#producer-3| [thrd:sasl_ssl://b-2.xxxxxxxxxxx.amazonaws.com:90]: sasl_ssl://b-2.xxxxxxxxxxxxx.amazonaws.com:9098/2: SASL authentication error: [38ed641d-4965-4af3-a12e-8482c24fef6c]: Access denied (after 220ms in state AUTH_REQ)