Propagate SSL settings to curl's SSL context

[lpsinger]

trafficstars

• Fixes #3751.
• Fixes confluentinc/confluent-kafka-python#1702.
• Fixes confluentinc/confluent-kafka-dotnet#2106.

Please note, I need some help to update the unit tests. Adding an rd_kafka_t * to the rdhttp functions means that the unit tests require at least a partly initialized rd_kafka_t struct.

Merging this will allow us to fix a very annoying error that our users are seeing (https://github.com/nasa-gcn/gcn-kafka-python/issues/49) by using librdkafka's builtin OIDC token refresh callback (https://github.com/nasa-gcn/gcn-kafka-python/pull/22).

[confluent-cla-assistant[bot]]

:tada: All Contributor License Agreements have been signed. Ready to merge.
:white_check_mark: lpsinger
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[emasab]

Hi @lpsinger thanks for this contribution, we're planning to add all SSL related configuration properties for libcurl and we're definitely considering this PR.

[lpsinger]

Hi @lpsinger thanks for this contribution, we're planning to add all SSL related configuration properties for libcurl and we're definitely considering this PR.

Should I rebase it?

[emasab]

It's not needed at the moment, I'll ask it when it's necessary as we aren't starting to work on this item very soon.

[lpsinger]

Just checking on this again. Are you ready to review this? Should I rebase it?

[anchitj]

Hi @lpsinger thanks for this contribution. I'm working on some more changes on top of your commit here https://github.com/confluentinc/librdkafka/pull/5077 to add a new config property for CA location.

[lpsinger]

Hi @lpsinger thanks for this contribution. I'm working on some more changes on top of your commit here #5077 to add a new config property for CA location.

Great. Would it help now if I rebased my changes?

[anchitj]

@lpsinger Not needed, I'll rebase them in #5077.

[tmancill]

+1 to getting this into a released version of librdkafka. In addition to fixing https://github.com/nasa-gcn/gcn-kafka-python/, it will address https://github.com/vectordotdev/vector/issues/21605#issuecomment-2459743411 and Vector Kafka sinks and sources will work with OIDC.

Maybe even out of the box instead of having to build... The only current work-around is to build locally from patched sources and incorporate that local build all the way up the dependency chain.

[chenlang-china-vo]

i use gcn_kafka to Subscribe to GCN Circular，when SASL authentication error occurs: {"status": "invalid_token"}, I will not receive any circular unless I restart my program. Do you have any good solutions? Will subscribing to GCN.Heartbeat solve this problem?

[lpsinger]

i use gcn_kafka to Subscribe to GCN Circular，when SASL authentication error occurs: {"status": "invalid_token"}, I will not receive any circular unless I restart my program. Do you have any good solutions? Will subscribing to GCN.Heartbeat solve this problem?

@chenlang-china-vo, I suggest that you open an issue over at https://github.com/nasa-gcn/gcn-kafka-python.

[emasab]

Thanks for the PR @lpsinger we preferred to have a separate configuration for HTTPS and SSL Kafka as using the Kafka truststore for HTTPS could cause certificate validation failures, so we went with something compatible with existing behavior that is using the dynamically linked OpenSSL path or for static linking using the manylinux path where the static binaries are compiled, that is also the first one in the probing function we reused (/etc/pki/tls/certs/ca-bundle.crt).

It was merged with https://github.com/confluentinc/librdkafka/pull/5133

[lpsinger]

Thank you!