[bitnami/keycloak] Ingress TLS config does not work and is badly documented

[The-Judge]

Name and Version

bitnami/keycloak 18.3.4

What architecture are you using?

amd64

What steps will reproduce the bug?

We deploy the Helm chart in the written version and pass the custom values to it as shown below.

We are using an nginx Ingress, which should handle the edge communication and SSL handling. We have bought "real"/static SSL certificates for our domain (replaced by example.com here).

We find the documentation on how to use this very unclear and can't find out the "correct" way:

1. The values.yaml file has a comment telling to "Please see README.md for more information" for ingress.secrets. The README.md says, that for this scenario where there are certs and keys available already, it's enough to create a secret named like the value used for ingress.hostname + -tls to get picked up automatically.
   This does not work at all.

2. Also, we tried to set the ingress.secrets section, using our whole certificate key and cert as described for that parameters.
   This results in the secret to be created just like described, but the Ingress will still use the K8s Dummy certificate instead of these.

Please assist either what's the correct approach in our case (having cert and key already) or make the documentation more clear.

Are you using any custom parameters or values?

production: true
proxy: edge

resources:
  limits:
    memory: 2Gi
  requests:
    cpu: 250m
    memory: 2Gi

ingress:
  enabled: true
  ingressClassName: nginx
  hostname: keycloak2-dev.example.com

  tls: true
  
  secrets:
    - name: "keycloak2-dev.example.com-tls"
      key: -----BEGIN PRIVATE KEY-----
        MIIEvgIBADANBgkqh ...REMOVED 
        -----END PRIVATE KEY-----
      certificate: -----BEGIN CERTIFICATE-----
        MIIGHjCCBQagAwIBA ...REMOVED 
        -----END CERTIFICATE-----

rbac:
  create: true
  rules:
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - get
    - list

autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 11
  targetCPU: ""
  targetMemory: ""

postgresql:
  enabled: true
  auth:
    postgresPassword: eo ...REMOVED
    username: keycloak
    password: wa...REMOVED
    database: keycloak
    existingSecret: ""
  architecture: standalone

What is the expected behavior?

No response

What do you see instead?

K8s Dummy certificates are used instead of provided certificate data.

Additional information

No response

[The-Judge]

Are there any ideas what else to try or any tendency for a fix already? We are stuck because of this and need to decide how to provide the customer with a fresh 28.* instance of Keycloak in the next couple of days.

[fmulero]

Hi @The-Judge

I tried to reproduce your issue without luck. I installed the chart version 18.4.0 with the following yaml (please pay attention about how the certificates are set):

resources:
  limits:
    memory: 2Gi
  requests:
    cpu: 250m
    memory: 2Gi

ingress:
  enabled: true
  ingressClassName: nginx
  hostname: keycloak2-dev.example.com

  tls: true
  secrets:
    - name: keycloak2-dev.example.com-tls
      key: |
        -----BEGIN PRIVATE KEY-----
        MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDo907W4/2UCRdq
        MKfWyynn8Cf8D88iFQg68zkUXY59hdtbnGkgLoJXa21FfqdYlASWKuVahKposMUW
        bfiMe1nUTNfKIi4e7dP4u39Kd4eDG2N8m+0VfsYluKCl9GGekcuMA26W/vMPokAu
        ...
        VGlOuXl8TlsM6Ei/mOUqR3I=
        -----END PRIVATE KEY-----
      certificate: |
        -----BEGIN CERTIFICATE-----
        MIIDTzCCAjegAwIBAgIUeHjkW1VX5b7OqLjWdTXLsizR4SgwDQYJKoZIhvcNAQEL
        BQAwWzELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
        GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEUMBIGA1UEAwwLZXhhbXBsZS5jb20w
        HhcNMjQwMjA5MDgxOTM3WhcNMjYxMTI5MDgxOTM3WjAkMSIwIAYDVQQDDBlrZXlj
        ....
        pHzUEY0gnJSjB9EPoFKl3v1wvhEiEONHStyziVuEdn4v6pqagysQEQhkWvUJ+erE
        uA4Xuo9tR3er/aTvefiKqefOfaxa+5NoolBzATM3NVxbmCA=
        -----END CERTIFICATE-----

This is the output of my console:

$ helm install keycloak oci://registry-1.docker.io/bitnamicharts/keycloak -f 23193.yaml 
NAME: keycloak
LAST DEPLOYED: Fri Feb  9 09:53:43 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: keycloak
CHART VERSION: 18.4.0
APP VERSION: 23.0.6

** Please be patient while the chart is being deployed **

Keycloak can be accessed through the following DNS name from within your cluster:

    keycloak.default.svc.cluster.local (port 80)

To access Keycloak from outside the cluster execute the following commands:

1. Get the Keycloak URL and associate its hostname to your cluster external IP:

   export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
   echo "Keycloak URL: https://keycloak2-dev.example.com/"
   echo "$CLUSTER_IP  keycloak2-dev.example.com" | sudo tee -a /etc/hosts

2. Access Keycloak using the obtained URL.

$ openssl s_client -connect keycloak2-dev.example.com:443 -servername keycloak2-dev.example.com -showcerts | openssl x509 
Warning: Reading certificate from stdin since no -in or -new option is given
depth=0 CN=keycloak2-dev.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=keycloak2-dev.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN=keycloak2-dev.example.com
verify return:1
-----BEGIN CERTIFICATE-----
MIIDTzCCAjegAwIBAgIUeHjkW1VX5b7OqLjWdTXLsizR4SgwDQYJKoZIhvcNAQEL
BQAwWzELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEUMBIGA1UEAwwLZXhhbXBsZS5jb20w
HhcNMjQwMjA5MDgxOTM3WhcNMjYxMTI5MDgxOTM3WjAkMSIwIAYDVQQDDBlrZXlj
....
pHzUEY0gnJSjB9EPoFKl3v1wvhEiEONHStyziVuEdn4v6pqagysQEQhkWvUJ+erE
uA4Xuo9tR3er/aTvefiKqefOfaxa+5NoolBzATM3NVxbmCA=
-----END CERTIFICATE-----

[The-Judge]

Hey @fmulero,

thank you very much for that snippet - we tried to apply this in our infrastructure with our certificates and it works! Since this is not too different from what we did there initially, we must have fallen for some temporary issue or unrecognized typo.

But there's another aspect to our understanding of the docs: What we did not understand at all is how to configure this chart so it uses an already existing secret that contains these certificates in K8s, so we do not have to enter the whole cert as plain text in our values file and end up with just another secret created, containing another copy of our certificates. Can you share how this can be done as well, please? We can't make sense of the docs on this topic.

[SemiCicada]

If you have a pre-existing TLS secret then you have to add an entry to the ingress.extraTls: part for the TLS portion of the ingress resource to be created. I also had trouble understanding the documentation.

[The-Judge]

@SemiCicada I'm sorry, but I don't get that info also. When I read the comments for extraTls, it looks like I need to provide a list of hostnames and the name of a secret. Just below that, there's ingress.secrets , which seems to do exactly the same. Which one to use in which situation?

[SemiCicada]

@The-Judge ingress.secrets I believe requires you to paste your secrets into the values file in plain text. By using ingress.extraTls you can specify a hostname and secret name so that you don't have to use plain text.

[The-Judge]

So, thanks for this fine example: There seem to be at least 3 ways to configure TLS with the chart and none of them is 100% clear in the docs:

1. ingress.secrets seems to take the full clear-text key with key and certificate chain with certificate.
2. ingress.extraTls seems to take a list of hosts and a secretName; the secret structure is explained at https://kubernetes.io/docs/concepts/services-networking/ingress/#tls . Looks quite straight forward, but the value comments state, this is intended "for additional hostnames to be covered". This indicates, that this is not designed to be used on its own and is intended to cover "additional" (more than 1) TLS in addition to some other ("main") TLS setup.

I have succeeded by the help of you guys and by a lot of try & error. But this should definately be explained better in the docs.

[fmulero]

Thanks a lot @SemiCicada for your help here.

I recognise that configure ingress TLS is not easy and I agree that the documentation can be easily improved. Also I think there is failure here because we shouldn't check ingress.secrets. If we remove that condition users could create their own TLS secrets following the name pattern explained in the documentation. I'll try to work on this.

Now that you faced these issues, feel free to open a PR with the changes you considere are needed, we will be happy to review it an learn from our mistakes.

[fmulero]

I've just created an internal task to improve the ingress configuration in the whole chart catalogue.

[busyboy77]

@The-Judge

If you can share the config used for adding custom secret. I'm facing the same issue and similarly getting confused with extraTLS etc.

Can you please share a working example here. Thanks

[The-Judge]

@The-Judge

If you can share the config used for adding custom secret. I'm facing the same issue and similarly getting confused with extraTLS etc.

Can you please share a working example here. Thanks

Hi @busyboy77 ,

this did the trick for me:

ingress:
  enabled: true
  ingressClassName: nginx
  hostname: keycloak.example.com

  tls: true

  extraTls:
    - hosts:
        - keycloak.example.com
      secretName: keycloak.example.com

https://kubernetes.io/docs/concepts/services-networking/ingress/#tls explains what the secret must look like.

[peeveen]

I just got caught out by this, and I think the main problem is that the documentation states:

If managing TLS secrets separately, it is necessary to create a TLS secret with name INGRESS_HOSTNAME-tls (where INGRESS_HOSTNAME is a placeholder to be replaced with the hostname you set using the *.ingress.hostname parameter).

... but there's no way to tell the chart that you've done this. And if you follow the extraTls solution, you can name the secret whatever you want. It doesn't have to follow that naming convention at all.

Ideally, in the ingress.yaml template, it needs another way of reaching the tls section. Perhaps introduce another value called e.g. ingress.existingSecret. It would be useful not to have to use the extraTls solution, as that creates a duplicated hostname configuration value, making ad-hoc configuration (using --set) more annoying.