Some Jenkins plugins could not be updated

[vittoriocanilli]

Describe the bug

I have setup my new Jenkins environment with the following Helm values file:

controller:
  installPlugins:
    - kubernetes:1.31.3
    - workflow-aggregator:2.6
    - git:4.10.3
    - configuration-as-code:1346.ve8cfa_3473c94

  additionalPlugins:
    - ansible:1.1
    - ansicolor:1.0.1
    - artifactory:3.15.3
    - authorize-project:1.4.0
    - azure-ad:191.vfc8019068670
    - azure-cli:0.9
    - basic-branch-build-strategies:1.3.2
    - blueocean:1.25.2
    - blueocean-jira:1.25.2
    - build-monitor-plugin:1.13+build.202201311821
    - build-user-vars-plugin:1.8
    - build-with-parameters:1.6
    - chromedriver:1.2
    - embeddable-build-status:2.0.3
    - extended-choice-parameter:0.82
    - extended-read-permission:3.2
    - external-monitor-job:191.v363d0d1efdf8
    - gatling:1.3.0
    - git-parameter:0.9.15
    - github-pullrequest:0.4.0
    - google-oauth-plugin:1.0.6
    - http_request:1.14
    - jacoco:3.3.1
    - locale:1.4
    - liquibase-runner:1.4.10
    - m2release:0.16.2
    - netsparker-cloud-scan:2.1.4
    - pam-auth:1.7
    - parameter-separator:1.3
    - parameterized-scheduler:1.0
    - parameterized-trigger:2.43
    - pipeline-utility-steps:2.12.0
    - pipeline-github-lib:36.v4c01db_ca_ed16	
    - pipeline-multibranch-defaults:2.1
    - saml:2.1.1-275.va_5718591a_999
    - slack:602.v0da_f7458945d
    - sonar:2.14
    - ssh-agent:1.24.1
    - ssh-slaves:1.33.0
    - thinBackup:1.10
    - timestamper:1.17
    - violation-comments-to-github:1.95
    - warnings-ng:9.11.0
    - windows-azure-storage:365.vf41653c43b01
    - ws-cleanup:0.40

  ingress:
    enabled: true
    apiVersion: "networking.k8s.io/v1"
    ingressClassName: nginx
    hostName:
      jenkins.devops.azure.example.com
    tls:
    - secretName: le-jenkins-devops-tls
      hosts:
        - jenkins.devops.azure.example.com

persistence:
  existingClaim: jenkins-devops-volume

Besides the requested Jenkins plugins, some other ones were installed as dependencies; three of them were installed with a version, which is not the latest one:

• JAXB installed 2.3.0 while the latest one is 2.3.0.1
• Oracle Java SE Development Kit Installer installed 1.0 while the latest one is 1.5
• WMI Windows Agents installed 1.0 while the latest one is 1.8

Therefore I have decided to add these 3 plugins with their latest versions explicitly to the Helm values:

...
  additionalPlugins:
    - ...
    - jaxb:2.3.0.1
    - jdk-tool:1.5
    - windows-slaves:1.8
    - ...
...

Unfortunately this did not help to update the 3 plugins. Moving all plugins to the installPlugins value did not help either.

Version of Helm and Kubernetes

- Helm: v3.7.2
- Kubernetes: v1.22.4 (client), v1.21.2 (server)

Chart version

jenkins-3.11.3

What happened?

1. Install the Helm chart (value files above and command below)
  1a. the complete value files requires that a certificate and a pvc are created in the same 
      Kubernetes namespace in advance (manifest files available below)
2. Verify whether all plugins are installed to their latest version
3. Add the not-updated plugins explicitly to the value files and upgrade the Helm chart
4. Verify again whether all plugins are installed to their latest version this time
5. Move all plugins to "installPlugins" in the value files and upgrade the Helm chart again
6. Verify once again whether all plugins are installed to their latest version now

What you expected to happen?

All plugins should have been updated.

How to reproduce it

helm install jenkins jenkins/jenkins \
  --namespace example-devops \
  --values helm/values.yaml \
  --set controller.tag="2.319.2-jdk11" \
  --version 3.11.3

(the values.yaml file can be found in the bug description above)

Anything else we need to know?

Here are the YAML manifest files of the certificate and the pvc, if necessary:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: le-jenkins-devops-tls
  namespace: example-devops
spec:
  secretName: le-jenkins-devops-tls
  dnsNames:
  - jenkins.devops.azure.example.com
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
    group: cert-manager.io

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-devops-volume
  namespace: example-devops
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: westeurope-1

This has been executed on an AKS (Azure) Kubernetes cluster.

[torstenwalter]

@vittoriocanilli Just a hint to avoid these kind of issues. Build your own Jenkins image and pre-install plugins there. That avoids many runtime issues.

[vittoriocanilli]

@vittoriocanilli Just a hint to avoid these kind of issues. Build your own Jenkins image and pre-install plugins there. That avoids many runtime issues.

Thanks @torstenwalter I have uninstalled the Helm chart and installed it again using a custom image with pre-installed plugins: nevertheless, the 3 plugins were not upgraded to their latest version, exactly like on my previous setup.

I have then uninstalled the Helm chart again and deleted the persistent volume claim (pvc), whose manifest file can be found in the bug's description above. After that I have created a new empty pvc and installed the chart with the custom image, I have got all plugins installed to their latest version.

I have then made a final test, where I have uninstalled the Helm chart and deleted the pvc once again; I have created another new empty pvc and installed the chart without the custom image and the plugins defined in additionalPlugins: even with this setup, the plugins are installed to their latest version.

I therefore presume there is a problem related to the pvc when updating some plugins. As you can imagine, I can not delete the pvc at every plugins update in a production environment. Do you have any suggestions?

[torstenwalter]

It's a misunderstanding. I meant to bundle all plugins in your custom Jenkins image and not to install any plugins as part of the Jenkins startup at all.

So all plugins you have now listed as additionalPlugins and installPlugins should be part of the image.

So additionalPlugins in your values.yaml should be empty and installPlugins: false.

See: https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#consider-using-a-custom-image

[vittoriocanilli]

That's exactly what I did.

[torstenwalter]

I have uninstalled the Helm chart and installed it again using a custom image with pre-installed plugins: nevertheless, the 3 plugins were not upgraded to their latest version, exactly like on my previous setup

You will end up with exactly those plugins which you have installed in your image. That's the whole point of it. The helm chart should not install or update any plugins in that setup. That's why you create an immutable image with a defined set of software (Jenkins version + specific plugin and version) baked into that image .

[joshbranham]

Correct me if I am wrong, but even if you install the plugins in a custom docker image, if you use persistence: from this chart, upon boot, the plugins will be stored forever and any changes to the plugins in the underlying docker image will be ignored (as they mount in /var/jenkins_home/plugins by default, and that folder is mapped to a volume when persistence is used)?

[vittoriocanilli]

@joshbranham if your question is addressed to me, I can confirm that I had the same impression that you have: the plugins installed in my custom Docker image seem to be ignored as I am using a persistent storage.

In my case, I am not letting the Helm chart create a pvc, but I let it use an existing pvc, which I have previously defined:

  persistence:
    enabled: true
    existingClaim: jenkins-devops-volume

You can find the manifest of the existing pvc in my issue's description above and my cluster runs on Azure AKS.

Nevertheless, after deleting the pvc and creating it back from scratch, I did not have further problems anymore. I keep this issue open in case I encounter the problem again though.

[joshbranham]

@joshbranham if your question is addressed to me, I can confirm that I had the same impression that you have: the plugins installed in my custom Docker image seem to be ignored as I am using a persistent storage.

In my case, I am not letting the Helm chart create a pvc, but I let it use an existing pvc, which I have previously defined:

  persistence:
    enabled: true
    existingClaim: jenkins-devops-volume

You can find the manifest of the existing pvc in my issue's description above and my cluster runs on Azure AKS.

Nevertheless, after deleting the pvc and creating it back from scratch, I did not have further problems anymore. I keep this issue open in case I encounter the problem again though.

So either there was a bug in the helm chart at some point that rendered this state, or some user action caused it. I suspect maybe installing a plugin via the UI is the switch that causes this failure?

[vittoriocanilli]

So either there was a bug in the helm chart at some point that rendered this state, or some user action caused it. I suspect maybe installing a plugin via the UI is the switch that causes this failure?

I really don't know what caused the problem: I did neither install nor update any plugin via the UI, as it is what I want to avoid when using this helm chart 😄