credentials.xml is kept from boot to boot even if the encryption keys seems to change, thus secrets inside becomes useless

[guilcy]

Jenkins and plugins versions report

Jenkins with casc on kubernetes, credentials strings are empty when requested by jenkins scripts

I am using jenkins inside kubernetes, I use casc to load the controler configuration. At first boot, everything is fine. At second boot, all credentials are lost. I noticed that the config.xml is regenerated at each boot, and that the hash of the secrets inside are different each time. Concerning the credentials.xml file, it is only generate one time at first boot. On second boot any credentials are empty when requested from the script console inside jenkins.

It is strange but it seems jenkins is regenerating the encryption keys at each boot. Thus, any configuration that is not newly generated contains hash that cannot be decoded by jenkins again. A I right ?

In addition I had a look to appy_config.sh (from init container), it is erasing all xml config file but not the credentials.xml... is it expected, shouldn't it be erased so that casc get a chance to create it newly (with its new encryption key) ?

Can you confirm the analyse is right: jenkins encryption key are generated again at each kubernetes deployment (each time I delete the pod) - I am not kubernetes specialist, sorry ? Why is credentials.xml not removed by apply_config.sh of the init container ?

Or maybe I miss something in the configuration. But all our JCASC settings were working fine inside a VM. Not sure why casc is not overring credentials.xml at each boot. But on VM it is performing a kind of merge if I remember well, I guess that because the keys are already defined it is not generating again with is weird, but that's another problem probably.

OS: Linux - 5.15.0-1073-azure
---
ace-editor:1.1
allure-jenkins-plugin:2.31.1
analysis-collector:2.0.0
analysis-core:1.96
analysis-model-api:12.1.0
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
artifactdeployer:1.3
asm-api:9.7-33.v4d23ef79fcc8
atlassian-bitbucket-server-integration:4.0.0
authentication-tokens:1.53.v1c90fd9191a_b_
azure-ad:507.vea_a_a_167b_d05c
azure-cli:0.9
azure-commons:1.1.3
azure-container-agents:253.vd2f5cd5c5040
azure-credentials:312.v0f3973cd1e59
azure-credentials-ext:1.0
azure-keyvault:251.vcfe31c013dc7
azure-sdk:174.va_89c1df897d2
badge:1.9.1
basic-branch-build-strategies:81.v05e333931c7d
batch-task:1.398.v550a_8e8ca_e5f
behave-testresults-publisher:0.0.11
bitbucket:241.v6d24a_57f9359
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jira:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-failure-analyzer:2.5.0
build-metrics:1.3
build-monitor-plugin:1.14-860.vd06ef2568b_3f
build-timeout:1.32
built-on-column:1.4
caffeine-api:3.1.8-133.v17b_1ff2e0599
categorized-view:1.13
cccc:0.6
checks-api:2.0.2
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
cmakebuilder:4.1.1
cobertura:1.17
code-coverage-api:4.99.0
codesonar:3.5.0
command-launcher:107.v773860566e2e
commons-compress-api:1.26.1-2
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
conditional-buildstep:1.4.3
config-file-provider:968.ve1ca_eb_913f8c
configuration-as-code:1775.v810dc950b_514
configurationslicing:548.ve92d48e66b_f8
console-tail:15.vc54cb_4ca_6981
copy-project-link:106.veb_028794a_844
copyartifact:722.v0662a_9b_e22a_c
coverage:1.13.0
cppcheck:1.26
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
csp:1.2
cucumber-perf:2.0.9
cucumber-reports:5.8.1
cucumber-testresult-plugin:0.10.1
custom-tools-plugin:0.8
cvs:2.19.1
dashboard-view:2.508.va_74654f026d1
data-tables-api:2.0.3-1
database:230.v701f20b_8b_f95
database-mysql:63.va_0596d2b_1438
disk-usage:1.2
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.4-86.v39b_a_5ede342c
docker-plugin:1.6
docker-workflow:572.v950f58993843
doxygen:178.v6ea_ef5f7dfdb
drmemory-plugin:1.12
dtkit-api:3.0.2
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:2.105
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
extended-choice-parameter:381.v360a_25ea_017c
extended-read-permission:53.v6499940139e5
extensible-choice-parameter:1.8.1
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
file-operations:214.v2e7dc7f25757
font-awesome-api:6.5.1-3
forensics-api:2.4.0
git:5.2.1
git-client:4.7.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1785.v99802b_69816c
gitlab-api:5.3.0-91.v1f9a_fda_d654f
gitlab-branch-source:703.vc5fd7effcc6b_
global-build-stats:293.vd7b_d6e361475
groovy-label-assignment:1.2.0
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.10.1-15.v0d99f670e0a_7
h2-api:11.1.4.199-12.v9f4244395f7a_
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
http_request:1.18
icon-shim:3.0.0
influxdb:3.6.1
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
ivy:2.5
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jenkins-multijob-plugin:627.v7c23cef20a_6a
jersey2-api:2.42-147.va_28a_44603b_d5
jira:3.13
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
job-dsl:1.87
job-import-plugin:3.6
job-restrictions:0.8
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
kubernetes:4203.v1dd44f5b_1cf9
kubernetes-cli:1.12.1
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:0.11
ldap:719.vcb_d039b_77d0d
leastload:3.0.0
lockable-resources:1246.v28b_e4cc6fa_16
logging:1.0.0
mail-watcher-plugin:1.19
mailer:472.vf7c289a_4b_420
mapdb-api:1.0.9-40.v58107308b_7a_7
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
memory-map:2.2.1 (53614)
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
momentjs:1.1.1
monitoring:1.98.0
multiple-scms:0.8
mysql-api:8.3.0-21.v2837a_a_360d57
nested-view:1.33
nexus-artifact-uploader:2.14
nexus-jenkins-plugin:3.15.438.vf87a_0dc45166
nodelabelparameter:1.12.0
oic-auth:4.371.vc7c0c06e8a_f5
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
openshift-client:1.1.0.424.v829cb_ccf8798
pam-auth:1.10
parameterized-scheduler:262.v00f3d90585cc
parameterized-trigger:787.v665fcf2a_830b_
patch-parameter:1.2
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-global-lib-nexus:0.3.2
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-maven:1396.veb_f07b_2fc1d8
pipeline-maven-api:1396.veb_f07b_2fc1d8
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:179.vc5cb_98f6db_38
plot:2.1.12
plugin-usage-plugin:4.4
plugin-util-api:4.1.0
pmd:4.0.0
popper-api:1.16.1-3
popper2-api:2.11.6-4
postbuild-task:1.9
prism-api:1.29.0-13
pubsub-light:1.18
pyenv-pipeline:2.1.2
rebuild:332.va_1ee476d8f6d
resource-disposer:0.23
rich-text-publisher-plugin:1.5
role-strategy:713.vb_3837801b_8cc
run-condition:1.7
saferestart:0.7
saml:4.464.vea_cb_75d7f5e0
scm-api:689.v237b_6d3a_ef7f
scoring-load-balancer:81.vb_82d531ff049
script-security:1326.vdb_c154de8669
shared-workspace:1.0.2
shiningpanda:0.24
short-workspace-path:0.3
show-build-parameters:1.0
sidebar-link:2.4.1
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
subversion:1256.vee91953217b_6
support-core:1427.v083f1d9372a_f
svnmerge:2.6
tasks:4.53
test-stability:2.3
text-file-operations:1.3.2
thinBackup:1.19
throttle-concurrents:2.14
timestamper:1.26
token-macro:400.v35420b_922dcb_
translation:1.16
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
virtualbox:0.7
warnings-ng:11.2.2
windows-azure-storage:419.v4046cd70d2e3
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-cps-global-lib:588.v576c103a_ff86
workflow-cps-global-lib-http:2.48.0
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45
xunit:3.1.4

What Operating System are you using (both controller, and any agents involved in the problem)?

jenkins/jenkins:2.440.1-jdk11

Reproduction steps

1. Start jenkins on kubernetes with a pvc for /var/jenkins_home, with jcasc defining some credentials (via some http server).
2. Dump the credentials from the console script, I can view all secrets
3. Delete the pod
4. Start again jenkins
5. Dump the credentials from the console script, secrets are empty strings

Expected Results

Step 5 should be like step 2

Actual Results

On step 5, credentials are empty string when requested from jenkins, even if the credentials.xml contains a hash for it. The hash is the same at each boot, but hash' in config.xml is changing at each boot.

Anything else?

Our secrets are stored in a Azure Key vault, we rely on jenkins casc mecanism to gather the secrets at boot time from the key vault. We are not using kubernetes secrets for that as, it is shared with existing jenkins on VM. The only solution I am thinking about is to add one line in the apply_config.sh, or another init container.

I know jdk11 is old, migration to a more recent jdk is planned as soon as this behavior is fixed (we are stuck today with this problem).