Possible compatibility issue with Confluent for Kubernetes

[perezjasonr]

Description

I believe librdkafka might not work with confluent's kafka brokers because you must have a truststore/keystore setup, which librdkafka does not have a truststore option (although it has keystore). And providing a CA cert does not seem to work.

How to reproduce

I followed these steps to set up confluent kafka on kubernetes:

https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/networking/external-access-static-host-based

I then used fluent bit, which uses librdkafka and has the same properties of librdkafka.

the producer cli shown in the example guide above, does work, however as you can see it has keystore and truststore in kafka.properties.

with fluentbit/librdkafka, i have tried:

key/cert/ca ssl protocol with just ca (no client key/cert) ssl protocol verify false ssl protocol with keystore and (since theres no truststore option), a CA to try to make up for it. plaintext (but I think CFK exposed externally won't allow this), just to try it, not that it'd be viable solution even if it worked

and I could never get through. I would have tried truststore but as you know that's not an option.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] librdkafka version (release number or git tag): its technically fluentbit v1.8.11 im trying to find out what version of librdkafka they are using, asking in the fluentbit open issue i made below
• [x] Apache Kafka version: latest confluent platform 7.1.1 brokers
• [x] librdkafka client configuration: paraphrased above, its basically just broker list and ssl properties
• [x] Operating system: containerized/kubernetes (but linux hosts)
• [x] Provide logs (with debug=.. as necessary) from librdkafka -- its not debug but i did provide logs, if we think it will be helpful fluentbit has the same debug options it passes to librd
• [x] Provide broker log excerpts -- see below where I provide link to open issue i made with fluentbit
• [x] Critical issue -- probably not critical but its pretty noteworthy if librdkafka cannot work with confluent platform

some findings/logs here:

https://github.com/fluent/fluent-bit/issues/5607

ive also gone ahead and tried to ask if basically confluent for kubernetes' kafka must be keystore and trust store and not some other combination that I tried mentioned above:

https://github.com/confluentinc/confluent-kubernetes-examples/issues/137

[MosheBlumbergX]

Hi @perezjasonr , you will need to pass your SSL config to the client side as outlined here https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka#configure-librdkafka-client

in the example above:

cat <<-EOF > $TUTORIAL_HOME/client/kafka.properties
bootstrap.servers=kafka.$DOMAIN:443
security.protocol=SSL
ssl.truststore.location=$TUTORIAL_HOME/client/client.truststore.p12
ssl.truststore.password=mystorepassword
ssl.truststore.type=PKCS12
ssl.keystore.location=$TUTORIAL_HOME/client/client.keystore.p12
ssl.keystore.password=mystorepassword
ssl.keystore.type=PKCS12
EOF

Should translate to something like this:

command_config = {'bootstrap.servers': 'kafka.$DOMAIN:443',
                  'security.protocol': 'SSL',
                  'ssl.ca.location': '$TUTORIAL_HOME/certs/cacerts.pem',
                  'ssl.certificate.location': '$TUTORIAL_HOME/certs/fullchain-client.pem',
                  'ssl.key.location': '$TUTORIAL_HOME/certs/privkey-client.pem',
                  'ssl.key.password': 'mystorepassword'
                  }

[perezjasonr]

Thank you for the response, I believe that was one of the scenarios I already tried:

with fluentbit/librdkafka, i have tried:

key/cert/ca

i also tried keystore with imported certs but since theres no truststore i tried a ca cert. It didnt take.

since I'm using fluentbit which uses librdkafka I did those same settings through fluentbit. this is why I'm of the impression that it wont work without a truststore at this time.

I think its also noteworthy that in the example and in the docs I can find no mention of non-truststore option (or a more standard pki setup) so i think this sort of confirms my suspicion:

https://docs.confluent.io/platform/current/kafka/encryption.html

there is no mention of a ca (as you put in the config ssl.ca.location), only truststore.

I have a feeling that either confluent's kafka needs to allow ssl.ca.location and regular cert/key configs, or librdkafka would need a truststore option for this to work, i saw keystore but no truststore:

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

this is why i say they are not compatible at this time. not saying librdkafka wouldnt work for other flavors of kafka: strimzi, bitnami, etc...but i think it wont work with confluent's, confluent for kubernetes (it might work with regular confluent platform though)

edit:

i will try again just for thoroughness and do exact 1:1 config options.

[OneCricketeer]

librdkafka wouldnt work for other flavors of kafka: strimzi, bitnami, etc

It can. It's literally all the same Apache Kafka being connected to.

think it wont work with confluent's, confluent for kubernetes (it might work with regular confluent platform though)

Again, still Apache Kafka...

The Confluent operator deploys Java based Docker images. Therefore, all those properties for the pods it manages are going to be JVM trust/key stores.

You can export or reuse certificates from JKS or pkcs12 files. Then you would mount/configure them for librdkafka, or non JVM based clients, such as sarama or kafka-python, etc

Regarding fluentbit, it accepts a conf object for all properties in the mentioned link

https://github.com/fluent/fluent-bit/blob/master/plugins/out_kafka/kafka_config.h#L105

[perezjasonr]

librdkafka wouldnt work for other flavors of kafka: strimzi, bitnami, etc

It can. It's literally all the same Apache Kafka being connected to.

you misquoted me. I said "I'm not saying", its the inverse of what you interpreted.

think it wont work with confluent's, confluent for kubernetes (it might work with regular confluent platform though)

Again, still Apache Kafka...

The Confluent operator deploys Java based Docker images. Therefore, all those properties for the pods it manages are going to be JVM trust/key stores.

thats kindof my point. so the other side needs to also have keystores does it not?

You can export or reuse certificates from JKS or pkcs12 files. Then you would mount/configure them for librdkafka, or non JVM based clients, such as sarama or kafka-python, etc

And which conf item is that in librdkafka? please show me

Regarding fluentbit, it accepts a conf object for all properties in the mentioned link

https://github.com/fluent/fluent-bit/blob/master/plugins/out_kafka/kafka_config.h#L105

This doesn't show anything. All it shows is that the conf is brought in as is but there is no truststore option here

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

I challenge anyone to get this working, but replace the producer cli with librdkafka or fluentbit. the proof is in the pudding.

https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/networking/external-access-static-host-based

[OneCricketeer]

the other side needs to also have keystores does it not?

Client certificates are needed, yes.

which conf item is that in librdkafka? please show me... there is no truststore option here

ssl.ca.certificate.stores includes a name of Trust.

[perezjasonr]

the other side needs to also have keystores does it not?

Client certificates are needed, yes.

Of course but I was specifically asking if it needed a keystore, as in actual keystore file. This isn't confirmable or deniable yet, anything to the contrary is an assumption and not based on demonstration.

which conf item is that in librdkafka? please show me... there is no truststore option here

ssl.ca.certificate.stores includes a name of Trust.

Thats not for a keystore. Remember that for context I've already tried a CA which also includes a name on "trust". First we have to prove it works with a regular ca (I already tried and it didnt work) or "Windows Certificate stores to load CA certificates from". The next best thing you suggested was this:

https://github.com/confluentinc/confluent-kafka-dotnet/issues/941#issuecomment-493179506

Which I at least think is worth a try. But Ive never heard a success story of librdkafka to CFK or fluentbit to CFK.

[OneCricketeer]

asking if it needed a keystore, as in actual keystore file.

No, because OpenSSL doesn't have the same concept as JVM key & trust stores. It does use keys and certificates, and uses a certificate authority... Those are different properties that should be configured and provided on your own.

never heard a success story of librdkafka to CFK or fluentbit to CFK

Nothing unique to CFK needs to be set, IMO. Kubernetes shouldn't matter. Confluent Platform/Server shouldn't matter... There's people actively using librdkafka, or clients built over it, with regular Apache Kafka brokers with SSL listeners.

[perezjasonr]

asking if it needed a keystore, as in actual keystore file.

No, because OpenSSL doesn't have the same concept as JVM key & trust stores. It does use keys and certificates, and uses a certificate authority... Those are different properties that should be configured and provided on your own.

Of course it doesn't have the same concept thats been my point from the beginning and why i keep making the distinction. librdkafka allows you to set the latter and not the former. Also I'm sure everyone here has seen a scenario where docs said "you must convert to jks" or "you must convert from jks to pem/crt" for X to work.

Which again, ive already tried to provide a CA and which is why my question is very deliberate. What I have not tried is your suggestion above where you extract direct from the jks. Maybe that will work...maybe...

never heard a success story of librdkafka to CFK or fluentbit to CFK

Nothing unique to CFK needs to be set, IMO. Kubernetes shouldn't matter. Confluent Platform/Server shouldn't matter... There's people actively using librdkafka, or clients built over it, with regular Apache Kafka brokers with SSL listeners.

It shouldnt matter, keyword there is shouldn't but it already clearly does. For example. confluent folks admittingly told me plaintext wont be accepted with CFK and with ingress, you can do plaintext with regular kafka, as I'm sure you know, or other kafka flavors (strimzi, bitnami, w/e). So this is clearly not off the table. You would actually be surprised how often Kubernetes deployment for a given app/tool changes things just because its tuned differently and making assumptions, ive seen it time and time again so this wouldn't surprise me the least

In the end as Ive said, its not confirmed yet. And cannot be confirmed, what youre saying is an assumption. And it will remain so until someone has a success story of librdkafka (or something that uses it, fluentbit) to CFK...Even the confluent folks don't sound sure in my direct email correspondence. They are suggesting I try it out.

[perezjasonr]

Finally got the darn thing working. Following these exact steps in the example (did the istio appendix):

https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/networking/external-access-static-host-based

then for librdkafka I used the following:

      security.protocol: ssl
      ssl.certificate.location: "path/to/fullchain-client.pem"
      ssl.ca.location: "path/to/cacerts.pem"
      ssl.keystore.location: "path/to/client.keystore.p12"
      ssl.keystore.password: password

now messages are showing up in the topic