Fails to obtain OAuth2 token using CA certificate on Ubuntu while it works on MacOS and Windows

[thomasnal]

Description

The same SSL certificate provided to 'ssl.ca.pem' property as a verbatim string succeeds on MacOS and Windows but fails on Linux/Ubuntu to verify the Confluent Cloud kafka broker. Why?

The CA certificate is the Let's Encrypt ISRG Root X1.

How to reproduce

var string cacert = @"-----BEGIN CERTIFICATE-----
...base64content-pem-downloaded-from-lets-encrypt...
-----END CERTIFICATE-----
";

AdminClientConfig adminClientConfig = new AdminClientConfig
{
    BootstrapServers = _kafkaClusterOptions.BootstrapServers,
    SaslMechanism = SaslMechanism.OAuthBearer,
    SecurityProtocol = SecurityProtocol.SaslSsl,
    SslCaPem = cacert,
    SaslOauthbearerMethod = SaslOauthbearerMethod.Oidc,
    SaslOauthbearerTokenEndpointUrl = _kafkaClusterOptions.OAuthTokenEnpointUrl,
    SaslOauthbearerClientId = _kafkaOAuthAppOptions.ClientId,
    SaslOauthbearerClientSecret = _kafkaOAuthAppOptions.ClientSecret,
    SaslOauthbearerExtensions = $"logicalCluster={_kafkaClusterOptions.ClusterId},identityPoolId={_kafkaClusterOptions.IdentityPoolId}"
using var adminClient = new AdminClientBuilder(adminClientConfig).Build();

await adminClient.CreateTopicsAsync(new TopicSpecification[] {
    new TopicSpecification { Name = "mytopic", ReplicationFactor = 1, NumPartitions = 1 }
});

 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [BRKMAIN] [Debug] [[thrd::0/internal]: :0/internal: Enter main broker thread]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [BROKER] [Debug] [[thrd:app]: sasl_ssl://my.westeurope.azure.confluent.cloud:9092/bootstrap: Added new broker with NodeId -1]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [CONNECT] [Debug] [[thrd:app]: sasl_ssl://my.westeurope.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [BRKMAIN] [Debug] [[thrd:sasl_ssl://my.westeurope.azure.confluent.cloud:9092/boot]: sasl_ssl://my.westeurope.azure.confluent.cloud:9092/bootstrap: Enter main broker thread]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [CONNECT] [Debug] [[thrd:sasl_ssl://my.westeurope.azure.confluent.cloud:9092/boot]: sasl_ssl://my.westeurope.azure.confluent.cloud:9092/bootstrap: Received CONNECT op]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [STATE] [Debug] [[thrd:sasl_ssl://my.westeurope.azure.confluent.cloud:9092/boot]: sasl_ssl://my.westeurope.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [INIT] [Debug] [[thrd:app]: librdkafka v2.2.0 (0x20200ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x46)]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
Error: ConfluentKafka : [rdkafka#producer-1] [OIDC] [Error] [[thrd:background]: Failed to retrieve OIDC token from "https://sso-dev.my.com/as/token.oauth2": error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt (-1)]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
Error: ntKafka : [rdkafka#producer-1] [ERROR] [Error] [[thrd:background]: Failed to acquire SASL OAUTHBEARER token: error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt]
 dbug: Data.Api.Tests.EndToEnd.Drivers.KafkaDriver[0]
       ConfluentKafka : [rdkafka#producer-1] [CONNECT] [Debug] [[thrd:main]: Cluster connection already in progress: no cluster connection]

Ubuntu ca-certificates bundle not needed since the PEM is provided as string.

After trying to supply a variety of values into SslCaLocation and SslCaPem I it appears that rdkafka returns this confusing error when it is not able to find a correct certificate in any of the supplied options. Note however that the PEM is correct - the code above works on MacOS and on Windows machines.

Anyone has an idea what is wrong on Linux/Ubuntu or can advise what to check?

Checklist

Please provide the following information:

• [x] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version.

   Top-level Package                                              Requested   Resolved
   > Confluent.Kafka                                              2.2.0       2.2.0

• [x] Apache Kafka version.

2.2.0

• [x] Client configuration.
• [x] Operating system.

Linux fv-az357-697 5.15.0-1041-azure #48-Ubuntu SMP Tue Jun 20 20:34:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• [x] Provide logs (with "debug" : "..." as necessary in configuration).
• [x] Provide broker log excerpts.
• [x] Critical issue.

[thomasnal]

Anyone can help with this problem? Librdkafka unreasonably tries to set some tls certificate while it is not asked to do it. Certificate is provided as a string via ssl.ca.pem option.

Librdkafka behaves the same way in python via confluent-kafka-python.

This remains a problem.

[dakodakov]

Hello,

I have a similar issue and found more information and a workaround in: https://github.com/confluentinc/librdkafka/issues/3751

[nhaq-confluent]

@thomasnal did the workaround provided in the previous comment help at all?

[thomasnal]

@nhaq-confluent Yes, the link to the issue on librdkafka helped to understand the situation. The issue is due librdkafka, since it is not able handle ca.location in this case.