VM provisioning fail using JCasC (`agentWorkspace` missing)

[pnag90]

Jenkins and plugins versions report

Environment

Jenkins: 2.479.3
OS: Linux - 6.8.0-1020-azure
Java: 17.0.9 - Azul Systems, Inc. (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:5.0.0
analysis-model-api:12.9.1
ansicolor:1.0.5
ant:511.v0a_a_1a_334f41b_
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.4-135.v4da_349961256
artifactory:4.0.8
asm-api:9.7.1-97.v4cc844130d97
authentication-tokens:1.119.v50285141b_7e1
authorize-project:2.0.0
aws-credentials:242.v8d05df911e3d
aws-java-sdk-ec2:1.12.772-477.v650d756dcf6d
aws-java-sdk-minimal:1.12.772-477.v650d756dcf6d
aws-java-sdk2-core:2.30.5-19.va_fd426c59215
aws-java-sdk2-ec2:2.30.5-19.va_fd426c59215
azure-ad:531.v13107da_f2635
azure-credentials:343.vd80f9c4859df
azure-keyvault:251.vcfe31c013dc7
azure-sdk:191.v53ec8913ee10
azure-vm-agents:975.va_a_d9b_6c912b_a_
badge:2.5
basic-branch-build-strategies:190.v343a_ee70d920
blueocean:1.27.16
blueocean-bitbucket-pipeline:1.27.16
blueocean-commons:1.27.16
blueocean-config:1.27.16
blueocean-core-js:1.27.16
blueocean-dashboard:1.27.16
blueocean-display-url:2.4.3
blueocean-events:1.27.16
blueocean-git-pipeline:1.27.16
blueocean-github-pipeline:1.27.16
blueocean-i18n:1.27.16
blueocean-jwt:1.27.16
blueocean-personalization:1.27.16
blueocean-pipeline-api-impl:1.27.16
blueocean-pipeline-editor:1.27.16
blueocean-pipeline-scm-api:1.27.16
blueocean-rest:1.27.16
blueocean-rest-impl:1.27.16
blueocean-web:1.27.16
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1208.vf528356feca_4
build-discarder:158.vce570d01ce4c
build-timeout:1.35
build-user-vars-plugin:182.v378b_9f14b_487
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.1
cloud-stats:377.vd8a_6c953e98e
cloudbees-bitbucket-branch-source:934.3.2
cloudbees-folder:6.980.v5a_cc0cb_25881
cloudbees-jenkins-advisor:358.v58972d19b_1f0
command-launcher:118.v72741845c17a_
commons-compress-api:1.26.1-2
commons-httpclient3-api:3.1-3
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.13.0-153.v91dcd89e2a_22
config-file-provider:982.vb_a_e458a_37021
configuration-as-code:1915.vcdd0a_d0d2625
copyartifact:761.vea_2b_25523e84
credentials:1405.vb_cda_74a_f8974
credentials-binding:687.v619cb_15e923f
data-tables-api:2.1.8-1
dependency-check-jenkins-plugin:5.6.0
display-url-api:2.209.v582ed814ff2f
docker-commons:445.v6b_646c962a_94
docker-workflow:592.v1001d948426c
durable-task:581.v299a_5609d767
ec2:1822.v87175d209b_b_5
echarts-api:5.5.1-5
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1876.v28d8d38315b_d
emoji-symbols-api:16.0-26.v9818ff7423f0
envinject:2.922.v2274200dd258
envinject-api:1.235.va_14c74f8f487
extended-read-permission:61.vf24570ff3b_e9
favorite:2.225.v68765b_b_a_1fa_3
file-operations:331.v9d88a_0206e2e
file-parameters:339.v4b_cc83e11455
folder-auth:217.vd5b_18537403e
font-awesome-api:6.6.0-2
forensics-api:2.7.0
git:5.7.0
git-client:6.1.1
git-parameter:0.10.0
github:1.41.0
github-api:1.321-478.vc9ce627ce001
github-branch-source:1810.v913311241fa_9
github-checks:589.v845136f916cd
gradle:2.14
groovy-postbuild:272.v52a_03efb_8653
gson-api:2.11.0-85.v1f4e87273c33
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.37
http_request:1.20
influxdb:4.0
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:310.v032f3f16b_0f8
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.16
jersey2-api:2.45-154.v4ded3dc34f81
jfrog:1.5.8
jira-steps:2.0.165.v8846cf59f3db
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jobConfigHistory:1296.v3459d8323cb_f
joda-time-api:2.13.0-93.v9934da_29b_a_e9
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-118.v7f23ed82a_8b_8
junit:1312.v1a_235a_b_94a_31
kubernetes:4306.vc91e951ea_eb_d
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
ldap:776.vddf3e325103b_
mailer:489.vd4b_25144138f
matrix-auth:3.2.3
matrix-project:845.vffd7fa_f27555
maven-plugin:3.24
metrics:4.2.21-461.v881e35d8fa_b_a_
mina-sshd-api-common:2.14.0-143.v2b_362fc39576
mina-sshd-api-core:2.14.0-143.v2b_362fc39576
mina-sshd-api-scp:2.14.0-143.v2b_362fc39576
next-build-number:66.v4b_4762172d53
node-iterator-api:55.v3b_77d4032326
oic-auth:4.457.v8d7c72676cfe
okhttp-api:4.11.0-183.va_87fc7a_89810
pam-auth:1.11
parameter-separator:258.vc2a_4c51b_de4c
pipeline-build-step:551.v178956c49ef8
pipeline-github-lib:61.v629f2cc41d83
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:749.v70084559234a_
pipeline-input-step:508.v584c0e9a_2177
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2221.vc657003fb_d93
pipeline-model-definition:2.2221.vc657003fb_d93
pipeline-model-extensions:2.2221.vc657003fb_d93
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2221.vc657003fb_d93
pipeline-stage-view:2.34
pipeline-utility-steps:2.18.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
preSCMbuildstep:71.v1f2990a_37e27
prism-api:1.29.0-18
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
resource-disposer:0.25
role-strategy:743.v142ea_b_d5f1d3
scm-api:703.v72ff4b_259600
script-security:1369.v9b_98a_4e95b_2d
simple-theme-plugin:202.v6367d3dea_73b_
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.17.3
sse-gateway:1.27
ssh-credentials:349.vb_8b_6b_9709f5b_
ssh-slaves:3.1021.va_cc11b_de26a_e
sshd:3.330.vc866a_8389b_58
strict-crumb-issuer:2.1.1
structs:338.v848422169819
support-core:1553.v4dd20218b_cb_2
swarm:3.49
test-results-analyzer:0.4.1
testng-plugin:835.v51ed3da_fcc35
text-finder:1.31
thinBackup:2.1.1
timestamper:1.28
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.6
variant:60.v7290fc0eb_b_cd
versioncolumn:320.v6b_b_814ca_01f7
warnings-ng:11.12.0
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1363.v03f731255494
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4014.vcd7dc51d8b_30
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1498.v33a_0c6f3a_4b_4
workflow-multibranch:800.v5f0a_a_660950e
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:686.v603d058a_e148
workflow-support:944.v5a_859593b_98a_
ws-cleanup:0.48

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins 2.479.3 Configuration as Code Plugin 1932.v75cb_b_f1b_698d Azure VM Agents 75.va_a_d9b_6c912b_a_

Reproduction steps

1. Create VM agent template manually via Jenkins Ui
2. Run a job -> Works
3. Store the cloud agent template in CasC
4. Run a job -> Works
5. Restart Jenkins & reload vms agent templates from CasC
6. Run a job -> Fails

Expected Results

6. Run a job -> Works

Actual Results

openjdk full version "17.0.14+7-LTS"
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3261.v9c670a_4748a_9
Launcher: AzureVMAgentSSHLauncher
Communication Protocol: Standard in/out
This is a Unix agent
ERROR: Connection terminated
java.io.EOFException
	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2933)
	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3428)
	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:985)
	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:416)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
	at hudson.remoting.Command.readFrom(Command.java:141)
	at hudson.remoting.Command.readFrom(Command.java:127)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:80)

java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details.
	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:325)
Caused: java.util.concurrent.ExecutionException
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:885)
Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:914)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:837)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Anything else?

No response

Are you interested in contributing a fix?

No response

[timja]

Compare the XML config in the config.xml of the Jenkins home between what is being saved by JCasC and what is being saved in the UI.

[pnag90]

I check the config.xml, some parameters are not synced:

<agentWorkspace>/var/jenkins</agentWorkspace>                                 
<ephemeralOSDisk>false</ephemeralOSDisk>
<encryptionAtHost>false</encryptionAtHost>
<shutdownOnIdle>false</shutdownOnIdle>
<spotInstance>false</spotInstance>
<installGit>false</installGit>
<installMaven>false</installMaven>
<installDocker>false</installDocker>
<installQemu>false</installQemu>
<retentionTimeInMin>0</retentionTimeInMin>
<trustedLaunch>false</trustedLaunch>
<acceleratedNetworking>false</acceleratedNetworking>
<templateDisabled>false</templateDisabled>
<templateVerified>false</templateVerified>

So it looks like setting a custom agent workspace made the trick here (ref: #617)

[timja]

thanks for tracking that down