Import CA certificates

[trotro]

Is your feature request related to a problem? Please describe

If one is using a custom CA you need to add it to the ca-bundle in the container and the keytore used by Jenkins. As of now, it is recommended to do it by creating a custom image, where you import your CA during the build.

Describe the solution you'd like

It is common to set a configMap for your custom CA on the cluster's level using the config.openshift.io/inject-trusted-cabundle: 'true' label.

It is allowed to add configMaps for the agents definition inside the Helm values using agent.volumes. Can we have the same feature for the controller ?

As far as I know, we can only define volumes for the controller for persistence.

Describe alternatives you've considered

Building a custom image containing my custom CA.

Additional context

A similar request was made https://github.com/jenkinsci/helm-charts/issues/189 but at that time, the configMap volume was not mentionned.

[maxnitze]

There is also the possibility to do this with an init container. Here's an example we do in our setup:

We add the certificates as a secret to the cluster:

user@nb [~/dev/]
-> % ls jenkins/helm/files/certificates 
myFirstCA.pem  mySecondCA.pem
user@nb [~/dev/]
-> % 

jenkins/helm/templates/secret-additional-certificates.yaml

{{- $files := .Files.Glob "files/certificates/*.pem" }}
{{- if $files }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ template "jenkins.fullname" .Subcharts.jenkins }}-additional-ca-certs
  namespace: {{ template "jenkins.namespace" .Subcharts.jenkins }}
  labels:
    "app.kubernetes.io/name": '{{ template "jenkins.name" .Subcharts.jenkins }}'
    {{- if .Values.jenkins.renderHelmLabels }}
    "helm.sh/chart": "{{ template "jenkins.label" .Subcharts.jenkins }}"
    {{- end }}
    "app.kubernetes.io/managed-by": "{{ .Release.Service }}"
    "app.kubernetes.io/instance": "{{ .Release.Name }}"
    "app.kubernetes.io/component": "{{ .Values.jenkins.controller.componentName }}"
stringData:
  {{- range $path, $fileContents := $files }}
  {{- $fileName := regexReplaceAll "(^.*/)(.*)\\.pem$" $path "${2}" }}
  {{ $fileName }}.pem: {{ $.Files.Get $path | toYaml | nindent 4 }}
  {{- end }}
{{- end }}

Then we mount them to a custom init container on startup, import them to the original cacerts file and mount that patched cacerts file later in the actual Jenkins container. We are using the same Jenkins image here, so we get the exact same cacerts file to start from.

jenkins/helm/values.yaml

jenkins:
  ...
  controller:
    ...
    customInitContainers:
      # updating the system and Java ca certs store requires the user to be root. so we do this in an init container and
      # mount the result to the running container. this way the actual Jenkins instance can run as a non-root user.
      - name: additional-ca-certs
        image: '{{ .Values.controller.image }}:{{- include "controller.tag" . -}}'
        imagePullPolicy: Always
        command:
          - sh
        args:
          - -c
          - |
            # add new ca certs to system store file
            cat /etc/ssl/certs/ca-certificates.crt /tmp/additional-ca-certs/*.pem > /ca-certificates/ca-certificates.crt

            # copy existing JVM `cacerts` file and add all additional ca certs
            cp ${JAVA_HOME}/lib/security/cacerts /ca-certificates/cacerts
            for file in /tmp/additional-ca-certs/*.pem; do
              filename=$(basename -- "$file" .pem)
              ${JAVA_HOME}/bin/keytool -import -trustcacerts -noprompt -keystore /ca-certificates/cacerts -storepass changeit -alias "$filename" -file "$file"
            done
        volumeMounts:
          - name: additional-ca-certs
            mountPath: /tmp/additional-ca-certs
          - name: ca-certificates
            mountPath: /ca-certificates
    ...
  persistence:
    ...
    volumes:
      - name: ca-certificates
        emptyDir: {}
      - name: additional-ca-certs
        secret:
          secretName: '{{ template "jenkins.fullname" . }}-additional-ca-certs'
    mounts:
      - mountPath: /etc/ssl/certs/ca-certificates.crt
        name: ca-certificates
        subPath: ca-certificates.crt
      - mountPath: /opt/java/openjdk/lib/security/cacerts
        name: ca-certificates
        subPath: cacerts
  ...

[pipelineRat]

I have tried the solution of an initContainer to add certs to the truststore. However I recieve permission denied from keytool when doing the import. All the containers are being run as user 1000.

keytool must be run as root AKA 0

Adding

securityContext:
  runAsUser: 0

doesnt work either. As the jenkins pod and all the containers, including init-containers just hang. Its like jenkins will not allow a root user even if it is just for an init-container

[maxnitze]

I guess you are trying to manipulate the original cacerts file? That will not work. That's why we copy it to a different place (in our case an emptyDir) and add the certificates there. Then in the original controller we mount the new file to the place, where it expects the ca-certificates file to be.

[pipelineRat]

            customInitContainers:
              - name: init-certs
                image: registry1.dso.mil/ironbank/redhat/openjdk/openjdk17:1.17
                command:
                  - sh
                args:
                  - -c
                  - |
                    cp /usr/lib/jvm/jre-17-openjdk/lib/security/cacerts /cacerts/cacerts
                    for f in /certs/*.crt;
                    do file=${f##*/}
                    alias=${file%.*}
                    echo $alias
                    keytool -cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias $alias -file $f
                    keytool -list -cacerts -alias $alias;
                    done;
                imagePullPolicy: IfNotPresent
                volumeMounts:
                  - name: cacerts
                    mountPath: /cacerts
                  - name: certs
                    mountPath: /certs

I am copying the cacerts to a new empty directory

[maxnitze]

I guess your keytool command misses the argument to change the cacerts file to use: -keystore /cacerts/cacerts. Your command will try to manipulate the original file ${JAVA_HOME}/lib/security/cacerts which it has no -access rights- write access to.

[pipelineRat]

Ill give that a try thankx maxnitze

[pipelineRat]

@maxnitze just replying back with what worked for me. even after changing my keystore pointer to -keystore /cacerts/cacerts i was still experiencing failures.

turns out my cp command was failing to create the copy of cacerts so I create my file /cacerts/cacerts first and then do a copy overrwrite of it.

            customInitContainers:
              - name: init-certs
                image: registry1.dso.mil/ironbank/redhat/openjdk/openjdk17:1.17
                command:
                  - sh
                  - '-c'
                  - |
                    touch /cacerts/cacerts;
                    yes | cp /usr/lib/jvm/jre-17-openjdk/lib/security/cacerts /cacerts/cacerts;
      
                    for f in /security/*.crt; do
                      file=${f##*/}
                      alias=${file%.*}
                      echo $alias
                      keytool -keystore /cacerts/cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias $alias -file $f
                      keytool -list -keystore /cacerts/cacerts -alias $alias;
                    done;
                imagePullPolicy: IfNotPresent
                volumeMounts:
                  - name: cacerts
                    mountPath: /cacerts
                  - name: certs
                    mountPath: /security