Question: What about SSL and SASL?

[Comradin]

Hello,

Looking at the chart I did not find any possible configuration for TLS encryption and SASL support.

Did you leave this out intentionally or do I miss something due to lack of experience?

Kind regards, Marcus

[narendraj9]

+1 for this question. I am also interested in having TLS support with the chart. For an installation like minikube, I think the chart can also make use of the k8s CA.

[matveybrant]

+1, working on adding this in my fork and will create a PR when ready - glad to have any help/feedback: https://github.com/wktmeow/cp-helm-charts/tree/ssl

[Arkoprabho]

@wktmeow Were you able to get this to work? Is there a way to configure SSL on Kafka using helm charts?

[matveybrant]

@Arkoprabho : I was able to get it working for local testing purposes, I've pushed my latest updates to my fork linked above. You'll need to install the chart from local dir, and you'll want to run certs.sh and client-certs.sh, updating the variables at the top of each script with your domain's specifics. I've also included my values.yaml that worked for me, but with my specific domain changed to 'domain.com'. Fill in your own settings as needed.

I haven't had much time to do more work on this so it's a little rough, and I haven't merged in the latest upstream changes. The two scripts I mentioned will generate the keystore files, and Helm will convert them to secrets for you. You can then use the client-certs.sh to generate additional client certs signed with the same CA, though if you wipe out the generated certs folder they will not be valid on future re-deploys.

[costa]

I've successfully set the following values

cp-kafka-rest:
  configurationOverrides:
    "ssl.truststore.location": "/var/private/ssl/kafka.client.truststore.jks"
    "ssl.keystore.location": "/var/private/ssl/kafka.client.keystore.jks"

only to discover I can't "volume"ize /var/private... damn

Any ideas as to how to do this with the official charts? Any estimation on actually being able to do this with the official charts?

Thanks all!

[costa]

@wktmeow thank you for your effort, looks really useful and thorough, however, as I see it, the only real showstopper is the inability to attach volumes (for the *stores); so, i'm going to try this tomorrow locally, and I'll appreciate any comments. Thanks again.

[matveybrant]

@costa could you elaborate on why you need to attach a volume for the stores? In my fork I'm having Helm create a secret from the jks files and then mount that secret as a volume. You could create a volume and then manually create the secrets in there and mount them in the deployments if you wanted, I suppose. Could also possibly add a kubernetes job in the helm chart that generates the keystores for you and then have those mounted from a volume. (Which, now that I think about it, may be a better pattern than what I have currently).

[costa]

@wktmeow thank you for your reply.

ok, a bit of context: at the moment, i'm interested in securing the rest proxy endpoint only, nothing else in the chart is going to be external and the internal communication is of much less interest from the security perspective to me right now. now i would like to have my owns means of control what goes in the "*store"s since i'd like to use the existing rest proxy capabilities for client authentication as well (the client keys shall be distributed to mobile clients). i'm relatively new to k8s, and i can't say i've completely got its spirit, so i don't really care if a "*store" goes through a secret before ending up a file in a rest proxy container, but i definitely think there should be an option for custom client+server key management, so although i welcome an option to have ssl out of the box with helm, i'm not sure how much flexibility should go into that solution (i'd just go for two extremes first: either everything is automatic with a single switch, or everything is custom through secrets or volumes or something).

i'm considering if it's worth the effort (in my case) to replace a pathetic workaround of building custom images with the "*store"s inside with your WIP PR where I don't need most of the stuff...

[matveybrant]

@costa Actually my WIP did not yet even include the HTTPS piece for any of the components, was going to get to it later but have been preoccupied with other work stuff. I did take a quick stab at it, if you'd like to have a look: https://github.com/wktmeow/cp-helm-charts/tree/rest-ssl

I haven't had a chance to test, but this should get you going if you create a certs folder at the root of the chart and add jks files that match the names you specify in values.yaml

Helm should then pull in the jks files, base 64 encode them and add them to (release-name)-rest-https-secret, which will then be mounted as a volume in the container in /etc/kafka/https. Feel free to shoot me a slack msg if you have questions, same username on Confluent slack 👍

I included a spot for all the https config options from: https://docs.confluent.io/current/kafka-rest/docs/config.html#kafka-rest-https-config

[costa]

@wktmeow thank you soo much! now the devil is apparently in the details, since i couldn't find anything in your branch i haven't tried myself. also, listeners is apparently essential and you branch doesn't have it yet. now i'm probably a little lost in the java universe, since I couldn't figure what was the deal with keytool not allowing different deststorepass and destkeypass, and the kafka-rest config demanding both. anyway, if i have my case working, i will share the generic parts, of course.

[costa]

Ok, I hope this gets resolved soon, meanwhile, my working "solution" of securing the rest proxy (https+auth) involves:

• creating the two JKS: https://gist.github.com/costa/f2cd91baf485f03bcee4de4c1fce7815
• pushing them into the kafka rest image: https://gist.github.com/costa/6b44941570f6a2ac86f4bd576dbd4a86
• deploying cp-helm-charts as a dependency or directly, while providing the passwords used above (through secrets and env vars or whatever): ... --set cp-helm-charts.cp-kafka-rest.configurationOverrides."ssl\.keystore\.password"="XXX",cp-helm-charts.cp-kafka-rest.configurationOverrides."ssl\.truststore\.password"="XXX",cp-helm-charts.cp-kafka-rest.configurationOverrides."ssl\.key\.password"="XXX",cp-helm-charts.cp-kafka-rest.configurationOverrides.listeners=https://0.0.0.0:8082,cp-helm-charts.cp-kafka-rest.configurationOverrides."ssl\.client\.auth"=true

This "solution" is not nearly good enough for any serious production mainly because it's not serious to build+push custom images just because you want to add/remove auth clients, but (a) it works; (b) i'm not sure that ...truststore.jks is the right way to go with many (really many) different clients.

[shadysoliman]

@wktmeow , I'm trying to deploy your https://github.com/wktmeow/cp-helm-charts/tree/rest-ssl, but I'm getting the following error: Error: YAML parse error on cp-helm-charts/charts/cp-kafka-rest/templates/deployment.yaml: error converting YAML to JSON: yaml: line 65: mapping values are not allowed in this context

Any idea what could be the issue ?

Regards Shady Soliman

[mmosttler]

I tried using your fork from @wktmeow for adding ssl. It seems it is failing to start the schema-registry due to EOF exception. Has anyone been able to get these charts loaded with ssl added or know how to resolve this issue?

io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryException: Error initializing the ssl context for RestService at io.confluent.kafka.schemaregistry.rest.SslFactory.<init>(SslFactory.java:67) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:121) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:58) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:73) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:40) at io.confluent.rest.Application.createServer(Application.java:201) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:42) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:653) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at io.confluent.kafka.schemaregistry.rest.SslFactory$SecurityStore.load(SslFactory.java:155) at io.confluent.kafka.schemaregistry.rest.SslFactory$SecurityStore.access$000(SslFactory.java:137) at io.confluent.kafka.schemaregistry.rest.SslFactory.createSslContext(SslFactory.java:86) at io.confluent.kafka.schemaregistry.rest.SslFactory.<init>(SslFactory.java:65) ... 6 more

thanks, mm

[DanielWozniak]

Hey @wktmeow , I tried pulling and running your repo but im having issues generating the secrets. When I run the helm charts as a whole I dont get no errors, the error shows up when I describe one of the pods which is stuck in container creating mode. The error is as follows:

Warning FailedMount 2m35s (x33 over 53m) kubelet, gke-ds-services-gke-dev-default-pool-b2c6dc03-zvmz MountVolume.SetUp failed for volume "ssl" : secrets "confluent-kafka-ssl-secret" not found

I checked and the secrets were not generated so I tried running the ssl-secret.yaml files individually with kubectl apply but it keeps complaining about about a colon.

error: error parsing ssl-secret.yaml: error converting YAML to JSON: yaml: line 11: found unexpected ':'

I got rid of the if statement since it had a syntax error with ne only having one parameter. {{- if and .Values.ssl.enabled (ne .Values.ssl.secretName) }}

So now my file looks like this:

#{{- if and .Values.ssl.enabled (ne .Values.ssl.secretName) }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ template "cp-kafka.ssl.secretName" . }}
  labels:
    app: {{ template "cp-ksql-server.name" . }}
    chart: {{ template "cp-ksql-server.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  kafka.client.truststore.jks: |-
{{- $path := printf "certs/%s" .Values.ssl.client.truststoreFile }}
{{ .Files.Get $path | b64enc | indent 4 }}
  kafka.client.keystore.jks: |-
{{- $path := printf "certs/%s" .Values.ssl.client.keystoreFile }}
{{ .Files.Get $path | b64enc | indent 4 }}
stringData:
  client.truststore.password: {{ .Values.ssl.client.truststorePassword }}
  client.keystore.password: {{ .Values.ssl.client.keystorePassword }}
  client.key.password: {{ .Values.ssl.client.keyPassword }}
#{{- end }}

Any suggestions what might be wrong with the code?

[reddy-s]

+1, Any updates on this?

[LiShangAn]

+1, Any resolves ?

[byomkesh99]

Any resolution on it, if anyone knows any link it would be a great help. Here I have one link, but not explained fully.

https://medium.com/weareservian/encryption-authentication-and-external-access-for-confluent-kafka-on-kubernetes-69c723a612fc

[Sukhdk]

Any update on this issue?

[mohankrishna3]

I have deployed this kafka helm chart in AKS. Now I am looking to implement SSL encryption and SSL Authentication with Azure AD for my project purpose. Could anyone help on configuring SSL encryption with my customized keystores and truststores ?

[romerorsp]

+1, In the end is SASL available to be configured for kafka helm chart? Any update?

[mak-454]

+1