Kubernetes Service Account within Jenkins credentials is missing in newly provisioned OC projects

[cschweikert]

Description:

After provisioning of a new project with the ODS provisioning app the Jenkins instance in the cd project does not contain a Kubernetes Service Account within Jenkins' global credentials store. If those credentials are missing, provisioning of new components is not working. The issue appeared after rebuilding the image jenkins-master:4.x from registry.redhat.io/openshift4/ose-jenkins:v4.8, ...:v4.9 or ...:v4.10.

Steps to reproduce the behavior:

1. Open the provisioning app and provision a new project including OC resources. Use default cd user or a project-specific user.
2. Wait until the project is provisioned and the Jenkins instance in the new cd namespace is up and running.
3. Open this Jenkins instance.
4. Open "Manage Jenkins > Manage Credentials".

The list now only contains the credentials for the chosen cd user, but the Kubernetes Service Account is missing.

Expected behavior:

Kubernetes Service Account should be there. Also its ID should match those that are in the credentials stores of Jenkins instances in other projects.

Affected version:

• OpenShift: 4.8
• OpenDevStack: 4.x
• jenkins-master: 4.x based on registry.redhat.io/openshift4/ose-jenkins:v4.8, ...:v4.9 and ...:v4.10

Log Output:

When trying to provision a component within the new project the Jenkins build hangs here:

[...]
Running on Jenkins in /var/lib/jenkins/jobs/...-cd/jobs/...-cd-prov-fe-angular-4-x/workspace
[Pipeline] {
[Pipeline] }
[Pipeline] // node
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘Jenkins’ doesn’t have label ‘...’

At the same time the following error shows up in the Jenkins pod's logs:

FATAL: Failed to connect to cloud openshift. There may be leftover resources on the Kubernetes cluster.
org.jenkinsci.plugins.kubernetes.auth.KubernetesAuthException: No credentials found with id <<ID of the Kubernetes Service Account>>

Additional context:

I did a lot of exploration on this issue already to narrow down the root cause. Most important to mention here is that this issue is only happening, when a Jenkins instance starts the first time in a newly provisioned project by using a jenkins-master:4.x image that was build from registry.redhat.io/openshift4/ose-jenkins:v4.8, ...:v4.9 or ...:v4.10. Rebuilding from ...:v4.7 and then provisioning a new project produces a Jenkins instance that has the right Kubernetes Service Account configured. Unfortunately ...:v4.7 causes yet another issue which might be related to https://github.com/opendevstack/ods-core/issues/1116.

Other findings are:

• Redeploying the Jenkins instance after the first start by using an image that was build from ose-jenkins:v4.7 or by using the image jenkins-master:3.x doesn't "heal" the issue.
• When first starting from ose-jenkins:v4.7 and then redeploy from image with ose-jenkins:v4.9 or so doesn't remove the credentials. Everything will get updated nicely. Also a manual update of plugins doesn't have any negative side effect.
• All ose-jenkins versions contain the same version of Jenkins 2.319.2.
• The ose-jenkins versions embed different version of certain plugins. Especially all of the Kubernetes-related plugins are different between ose-jenkins:v4.7 and those in ...:v4.8, ...:v4.9 or ...:v4.10 (see also https://github.com/openshift/jenkins/pull/1398/files#diff-aa0807faadc6d3a6dfa311d61c49d815baab61b756eb321b00dc3392fd7648b8R61).
• For the problem it doesn't matter, if the project was created with or without a project-specific cd user.
• Line-by-line comparing the log output of the project provisioning with recently provisioned projects doesn't bring significant differences. Also it doesn't matter, if they where provisioned with ODS version 3.x or 4.x.
• Jenkins sometimes is doing migrations of configs, credentials, etc. when a new version of a plugin is installed.

Assumption:

When looking at all the findings my assumption is that all of this comes from the following two aspects:

• one or more plugins are now preinstalled with newer versions:

kubernetes:1.31.0
kubernetes-client-api:5.10.1-171.vaa0774fb8c20
kubernetes-credentials:0.9.0

• Jenkins skipped a necessary migration from an even older version of a plugin OR the new versions don't support automatic import of the Kubernetes Service Account.

Does anybody know where this Kubernetes Service Account is actually coming from and how it gets provisioned into Jenkins credentials?

[cschweikert]

@braisvq1996 FYI

[BraisVQ]

We have updated Jenkins to latest available version V4.10 and did not find any issue with it either from a new instance or updating from V4.7 See #1235