Enhance the generation and handling of keystore and truststore

[leochr]

There are various ways to inject keystore and truststore for the application. In some cases, some conflicts could prevent the application from working as expected. Re-evaluate the generation and handling of keystore and truststore and improve as possible.

Open Liberty runtime also supports providing certificates as env var to establish trust. Switch to this mechanism if possible: https://openliberty.io/docs/latest/reference/feature/transportSecurity-1.0.html#_provide_certificates_from_an_environment_variable_or_a_file

[leochr]

@idlewis please add your analysis of truststore handling in Liberty container here. Thank you

[idlewis]

I think I can see two issues with the trust store handling in docker-server.sh

1. The script uses 'keytool -import' to add certificates to the default trust store in 2 situations (ca.crt exists in /etc/x509/certs or SEC_IMPORT_K8S_CERTS=true). If there is an existing trust store, this command will fail, as the passwords won't match
2. If a trust store exists in the default location, then the script will add its own config to config overrides. If there was an existing trust store before the script ran, this config will cause a password mismatch, and an error on Liberty startup.

The fix for 2 is pretty simple (only add config if docker-server.sh created the truststore)

For 1, we need to decide what to do in the case of an existing trust store.

[justin-fleming]

Summary

The docker-server.sh script shipped with Liberty does not import .crt files that are missing a newline at the end of the file in the /var/run/secrets/kubernetes.io/serviceaccount dir.

The problem is on this line where we are cating *.crt and attempting to iterate over the certs to import them: https://github.com/OpenLiberty/ci.docker/blob/main/releases/latest/full/helpers/runtime/docker-server.sh#L62

Steps to recreate

Setup:

• Given 2 cert files: thing1.crt and thing2.crt
• I simulated the customers 'bad' certs by doing the following:

  mkdir badcerts
  printf "%s" "$(< thing1.crt)" > badcerts/thing1-bad.crt
  printf "%s" "$(< thing2.crt)" > badcerts/thing2-bad.crt
  

  This removes the newline at the end of the files:

  # diff thing1.crt badcerts/thing1-bad.crt 
  19c19
  < -----END CERTIFICATE-----
  ---
  > -----END CERTIFICATE-----
  \ No newline at end of file
  

Without newlines at the end of the cert files, no truststore is generated (even though the certs are mounted):

• Created a configmap with the bad certs:

  oc create configmap --from-file cert1.crt=badcerts/thing1-bad.crt --from-file cert2.crt=badcerts/thing2-bad.crt certs
  

• Created this deployment:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: liberty-deployment
    namespace: cert-test
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: liberty-test
    template:
      metadata:
        labels:
          app: liberty-test
        namespace: liberty-test
      spec:
        containers:
        - name: liberty
          image: image-registry.openshift-image-registry.svc:5000/cert-test/ol-test:1.0
          ports:
          - containerPort: 9080
          imagePullPolicy: Always
          volumeMounts:
          - name: cert-volume
            mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          env:
          - name: SEC_IMPORT_K8S_CERTS
            value: 'true'
        volumes:
          - name: cert-volume
            configMap:
              name: certs
  

• Check details in the running pod:

  # oc rsh liberty-deployment-86df686575-8kc72 
  sh-4.4$ ls -la /output/resources/security/
  total 4
  drwxrwx---. 2 default root   21 Feb 15 19:56 .
  drwxrwx---. 1 default root   22 Feb 15 19:56 ..
  -rw-rwx---. 1 default root 2650 Feb 15 19:56 key.p12
  
  sh-4.4$ ls -la /var/run/secrets/kubernetes.io/serviceaccount/
  total 0
  drwxrwsrwx. 3 root 1000680000 94 Feb 22 17:21 .
  drwxr-xr-x. 3 root root       60 Feb 22 17:21 ..
  drwxr-sr-x. 2 root 1000680000 40 Feb 22 17:21 ..2024_02_22_17_21_26.2791951379
  lrwxrwxrwx. 1 root 1000680000 16 Feb 22 17:21 cert1.crt -> ..data/cert1.crt
  lrwxrwxrwx. 1 root 1000680000 16 Feb 22 17:21 cert2.crt -> ..data/cert2.crt
  lrwxrwxrwx. 1 root 1000680000 32 Feb 22 17:21 ..data -> ..2024_02_22_17_21_26.2791951379
  
  sh-4.4$ cat *.crt
  -----BEGIN CERTIFICATE-----
  ...
  cert-data-1
  ...
  -----END CERTIFICATE----------BEGIN CERTIFICATE-----
  ...
  cert-data-2
  ...
  -----END CERTIFICATE-----sh-4.4$
  

With newlines, the certs are imported into the truststore:

• Delete & recreate the configmap with the 'good' certs and restart the app pod:

  oc delete cm certs
  oc create configmap --from-file cert1.crt=thing1.crt --from-file cert2.crt=thing2.crt certs
  oc delete po -l 'app=liberty-test'
  

• Check details in the new pod:

  # oc rsh liberty-deployment-86df686575-s42f7
  sh-4.4$ ls -la /output/resources/security/
  total 8
  drwxrwx---. 1 default    root   23 Feb 22 17:35 .
  drwxrwx---. 1 default    root   22 Feb 15 19:56 ..
  -rw-rwx---. 1 default    root 2650 Feb 15 19:56 key.p12
  -rw-r--r--. 1 1000680000 root 2054 Feb 22 17:35 trust.p12
  sh-4.4$ cat /var/run/secrets/kubernetes.io/serviceaccount/*.crt
  -----BEGIN CERTIFICATE-----
  ...
  cert-data-1
  ...
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  ...
  cert-data-2
  ...
  -----END CERTIFICATE-----
  sh-4.4$