jenkinsci
Frequent exception "java.io.StreamCorruptedException: invalid stream header" reported by the pipeline using GCE plugin
Jenkins and plugins versions report
Environment
Jenkins: 2.492.1 OS: Linux - 5.14.0-503.21.1.el9_5.x86_64 Java: 17.0.14 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
ace-editor:1.1 active-directory:2.39 analysis-model-api:12.9.1 ansicolor:1.0.6 ant:511.v0a_a_1a_334f41b_ antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83 apache-httpcomponents-client-5-api:5.4-136.v5a_21779c63f8 artifactory:4.0.8 asm-api:9.7.1-97.v4cc844130d97 authentication-tokens:1.119.v50285141b_7e1 aws-credentials:243.v41c19a_fb_5dcf aws-java-sdk-ec2:1.12.772-477.v650d756dcf6d aws-java-sdk-minimal:1.12.772-477.v650d756dcf6d aws-java-sdk2-core:2.30.10-22.v1339d66b_e0f4 aws-java-sdk2-ec2:2.30.10-22.v1339d66b_e0f4 blueocean:1.27.16 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.16 blueocean-commons:1.27.16 blueocean-config:1.27.16 blueocean-core-js:1.27.16 blueocean-dashboard:1.27.16 blueocean-display-url:2.4.3 blueocean-events:1.27.16 blueocean-git-pipeline:1.27.16 blueocean-github-pipeline:1.27.16 blueocean-i18n:1.27.16 blueocean-jira:1.27.16 blueocean-jwt:1.27.16 blueocean-personalization:1.27.16 blueocean-pipeline-api-impl:1.27.16 blueocean-pipeline-editor:1.27.16 blueocean-pipeline-scm-api:1.27.16 blueocean-rest:1.27.16 blueocean-rest-impl:1.27.16 blueocean-web:1.27.16 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.80-256.vf98926042a_9b_ branch-api:2.1214.v3f652804588d build-name-setter:2.5.0 build-timeout:1.35 build-user-vars-plugin:195.v8c35f9d5c3dc caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.2 cloud-stats:377.vd8a_6c953e98e cloudbees-bitbucket-branch-source:934.4.2 cloudbees-folder:6.982.vf165a_16c9507 command-launcher:118.v72741845c17a_ commons-compress-api:1.27.1-2 commons-lang3-api:3.17.0-84.vb_b_938040b_078 commons-text-api:1.13.0-153.v91dcd89e2a_22 conditional-buildstep:1.5.0 config-driven-pipeline:1.3 config-file-provider:982.vb_a_e458a_37021 configuration-as-code:1932.v75cb_b_f1b_698d copyartifact:765.v0357cc6e6eb_3 credentials:1408.va_622a_b_f5b_1b_1 credentials-binding:687.v619cb_15e923f dashboard-view:2.528.v3470c02b_d7c9 data-tables-api:2.1.8-1 display-url-api:2.209.v582ed814ff2f docker-build-step:2.12 docker-commons:445.v6b_646c962a_94 docker-java-api:3.4.1-96.v77147a_de67f8 docker-plugin:1.9.0 docker-workflow:595.v60cec912059b_ durable-task:581.v299a_5609d767 echarts-api:5.5.1-5 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:1876.v28d8d38315b_d envinject:2.926.v69c9b_3896a_96 envinject-api:1.235.va_14c74f8f487 extended-choice-parameter:382.v5697b_32134e8 external-monitor-job:221.v35059272565b_ favorite:2.225.v68765b_b_a_1fa_3 font-awesome-api:6.6.0-2 forensics-api:2.7.0 generic-webhook-trigger:2.3.1 git:5.7.0 git-client:6.1.2 git-parameter:0.11.0 git-server:126.v0d945d8d2b_39 github:1.41.0 github-api:1.321-478.vc9ce627ce001 github-branch-source:1810.v913311241fa_9 github-pullrequest:0.7.2 gitlab-plugin:1.9.7 google-compute-engine:4.683.v0ce26579a_ee7 google-oauth-plugin:1.330.vf5e86021cb_ec gradle:2.14 groovy:497.v7b_061a_a_de65d gson-api:2.12.1-113.v347686d6729f h2-api:11.1.4.199-30.v1c64e772f3a_c handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.37 http_request:1.20 icon-shim:3.0.0 instance-identity:201.vd2a_b_5a_468a_a_6 ionicons-api:74.v93d5eb_813d5f ivy:2.9 jackson2-api:2.17.0-386.vcb_b_037da_0d62 jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:310.v032f3f16b_0f8 javax-activation-api:1.2.0-7 javax-mail-api:1.6.2-10 jaxb:2.3.9-1 jdk-tool:83.v417146707a_3d jenkins-design-language:1.27.16 jersey2-api:2.45-154.v4ded3dc34f81 jira:3.13 jjwt-api:0.11.5-112.ve82dfb_224b_a_d jnr-posix-api:3.1.20-125.vb_6ec4b_21b_15e job-dsl:1.90 job-import-plugin:3.6 jobConfigHistory:1302.va_65b_ccb_fde71 joda-time-api:2.13.1-115.va_6b_5f8efb_1d8 jquery:1.12.4-3 jquery-detached:1.2.1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20250107-125.v28b_a_ffa_eb_f01 json-path-api:2.9.0-148.v22a_7ffe323ce junit:1314.vd966e9a_88895 kubernetes:4306.vc91e951ea_eb_d kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials:190.v03c305394deb_ ldap:776.vddf3e325103b_ localization-support:1.2 localization-zh-cn:371.v23851f835d6b_ lockable-resources:1349.v8b_ccb_c5487f7 mailer:489.vd4b_25144138f mapdb-api:1.0.9-40.v58107308b_7a_7 matrix-auth:3.2.4 matrix-project:845.vffd7fa_f27555 maven-plugin:3.25 mercurial:1309.v6802b_f0efb_b_9 metrics:4.2.21-461.v881e35d8fa_b_a_ mina-sshd-api-common:2.14.0-143.v2b_362fc39576 mina-sshd-api-core:2.14.0-143.v2b_362fc39576 momentjs:1.1.1 nodejs:1.6.3 oauth-credentials:0.653.v14cf2088e950 okhttp-api:4.11.0-183.va_87fc7a_89810 oss-symbols-api:296.v4981240eeb_1a_ pam-auth:1.11 parameterized-trigger:840.v3c7d4a_a_5e6c7 pipeline-build-step:555.v589d5c24a_3d6 pipeline-github-lib:61.v629f2cc41d83 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-groovy-lib:752.vdddedf804e72 pipeline-input-step:513.v7eb_fe7e2c0fc pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2221.vc657003fb_d93 pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:2.2221.vc657003fb_d93 pipeline-model-extensions:2.2221.vc657003fb_d93 pipeline-rest-api:2.37 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2221.vc657003fb_d93 pipeline-stage-view:2.37 pipeline-utility-steps:2.18.0 plain-credentials:183.va_de8f1dd5a_2b_ plugin-util-api:5.1.0 popper-api:1.16.1-3 popper2-api:2.11.6-5 prism-api:1.29.0-18 publish-over:0.22 publish-over-ssh:387.vec3df0f668cd pubsub-light:1.18 rebuild:338.va_0a_b_50e29397 resource-disposer:0.25 role-strategy:756.v978cb_392eb_d3 run-condition:243.v3c3f94e46a_8b_ scm-api:703.v72ff4b_259600 script-security:1373.vb_b_4a_a_c26fa_00 simple-queue:1.4.6 slack:761.v2a_8770f0d169 snakeyaml-api:2.3-123.v13484c65210a_ sonar:2.18 sse-gateway:1.27 ssh:158.ve2a_e90fb_7319 ssh-agent:376.v8933585c69d3 ssh-credentials:349.vb_8b_6b_9709f5b_ ssh-slaves:3.1031.v72c6b_883b_869 sshd:3.353.v2b_d33c46e970 structs:338.v848422169819 subversion:1287.vd2d507146906 thinBackup:2.1.2 throttle-concurrents:2.16 timestamper:1.28 token-macro:444.v52de7e9c573d trilead-api:2.147.vb_73cc728a_32e uno-choice:2.8.6 variant:70.va_d9f17f859e0 windows-slaves:1.8.1 workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1363.v03f731255494 workflow-basic-steps:1079.vce64b_a_929c5a_ workflow-cps:4018.vf02e01888da_f workflow-cps-global-lib:612.v55f2f80781ef workflow-durable-task-step:1405.v1fcd4a_d00096 workflow-job:1505.vea_4b_20a_4a_495 workflow-multibranch:800.v5f0a_a_660950e workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:686.v603d058a_e148 workflow-support:946.v2a_79d8a_4b_e14 ws-cleanup:0.48
What Operating System are you using (both controller, and any agents involved in the problem)?
Controller - Red Hat Enterprise Linux release 9.5 Slave - GCP VM based on Debian 12.
Reproduction steps
Trigger multiple pipelines in parallel. Each of these pipelines provision a dynamic Vm and run pipeline stages. Some stages are on the host and other run inside the docker container.
Expected Results
No abnormal termination of the pipelines.
Actual Results
Pipeline are terminated randomly with the below exception.
java.io.StreamCorruptedException: invalid stream header: 636F7272
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:958)
at java.base/java.io.ObjectInputStream.
Anything else?
No response
Are you interested in contributing a fix?
No response
@mkjkec2005 ,
This issue might be due to an incompatibility between the agent and controller, possibly related to the Java version.
Relevant references:
I see you're running the latest controller with Java 17 (correct setup). Are all agents also on Java 17? If a project requires an older version, the agent should have multiple Java versions installed, with the agent process itself running on Java 17.
This doesn't seem directly related to the GCE plugin but might be influenced by pipeline design, especially with multiple agents and parallel stages. If you can provide a minimal reproducible pipeline, I'd be interested to take a closer look.
Also, the pipeline uses all dynamic agents or are there any static agents involved? if there are static agents - the version of the agent should be compatible with the controller (i.e. download the agent.jar from the controller itself), to avoid any remoting version compatibility issues.
@gbhat618 Thank you for your response, and apologies for the delayed reply as I was out of the office. We are using a simple pipeline that runs a few shell scripts, which takes around 1 to 1.5 hours to complete. Please note that it fails roughly once out of every 10 runs. For your reference, I’ve included the pipeline snippet below. We rely heavily on this plugin, and this issue is significantly affecting the reliability of our automation.
`pipeline { agent { label 'my-gce-pool' }
options {
timestamps()
buildDiscarder(logRotator(numToKeepStr: '20'))
timeout(time: 3, unit: 'HOURS')
skipDefaultCheckout()
}
environment {
BUILD_WORKSPACE = "<some workspace>"
}
stages {
stage('Setup workspace') {
steps {
script {
// some git clone commands
}
}
}
stage('Model Validation') {
steps {
script {
sh '''
// export some shell variables
export CHANGE_ID=${PULL_REQUEST_ID}
export CHANGE_TARGET=${PULL_REQUEST_TO_BRANCH}
echo "CHANGE_ID is $CHANGE_ID"
echo "CHANGE_TARGET is $CHANGE_TARGET"
BIN_DIR=${BUILD_WORKSPACE}/bin
YANG_DIR=${BUILD_WORKSPACE}/yang
PYANG_DIR="../update_pyang"
//execute some shell commands that take totally 1-1.5 hr to complete.
'''
}
echo "stage completed!!!"
}
}
}
}
post {
always {
echo 'Cleanup workspace!!!'
sh 'sudo rm -rf ${BUILD_WORKSPACE}'
}
}
`
I suffered 'random' java.io.StreamCorruptedException: invalid stream header: 636F7272 errors for a couple of months.
I was using GCE plugin with an instance template based on debian.
- Created in early Jan'25
- Test ran fine until mid to late Jan, what I thought was random
....invalid stream header: 636F7272 - JDK was being updated. ** Checked JDK updates on debian, and there was a minor update rolled out!
My issue issue was entirely down to unattended-upgrades being enabled on the instance I was using. The default JDK was getting updated and pulling the rug from under the Jenkins agent, BANG that exception every time!
Since disabling unattended-upgrades I've had no issues.
- If you are concerned about disabling because of security, then simply recreate a new template or instance that the GCE plugin references created with latest versions. Update when you need security patches, but ensure the floor under you is solid and not changing.
It may not be your issue but, it was impacting our environment.
Anyway, hope it may help.
Thank you @smithdm. I have made the change to disable unattended-upgrades. Let me monitor it for couple of days and share the observation.
@smithdm Was the "invalid stream header" completely eliminated after disabling unattended upgrades in your case? In the last 24 hours, I observed that the frequency of this random error is less, but I just hit the same one again.
java.io.StreamCorruptedException: invalid stream header: 636F7272 at java.base/java.io.ObjectInputStream.readStreamHeader(Unknown Source) at java.base/java.io.ObjectInputStream.<init>(Unknown Source) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to gx-dcc-pool-7z0k8l at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:1002) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121) at hudson.Launcher$ProcStarter.start(Launcher.java:506) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:314) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:295) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:292) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:185) at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:111)
This mostly happens when a pipeline stage running under a docker container ends and the Docker plugin tries to stop the container.
[Pipeline] } 07:13:50 $ docker stop --time=1 ae53a585ebd5f38636d041905e867a1097ce4232708b92a8fd7047993518e5b5 --> Seen after this. [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // stage
@smithdm , @gbhat618 Any clue on this issue? Any pointers will be really helpful. Thank you. java.lang.InterruptedException at java.base/java.lang.Object.wait0(Native Method) at java.base/java.lang.Object.wait(Unknown Source) at hudson.remoting.Request.call(Request.java:177) at hudson.remoting.Channel.call(Channel.java:1002) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121) Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 16ed9dae-d5e9-4b14-908a-9fd6c9a64b94 Caused: java.io.InterruptedIOException This mostly happens when a pipeline stage running under a docker container ends and the Docker plugin tries to stop the container. We are using reuseNode in the pipeline agent.
The above stacktrace is not sufficient to say where the error occurred.
We are using reuseNode in the pipeline agent.
The pipeline sample you shared last week doesn't include docker-plugin or the reuseNode part, so it's unclear which one might be causing the issue.
This discussion has limited visibility. I suggest posting on Jenkins Community, where more experts on pipeline design can help identify the issue.
By any chance, do you have two Jenkins instances pointing to the same docker infra (i.e. same config in docker cloud in jenkins)? (perhaps you may have cloned the Jenkins instance for testing purposes or something..)
There is an orphan docker container cleanup in the docker-plugin, which runs every 5min -- it will cleanup the container if those container are not attached to the current Jenkins instance. If that is the case - a way to prevent that kind of cleanup happening is by passing a system property during jenkins startup (you would need to do that for both jenkins if they are pointing to same docker infra)
-Dcom.nirima.jenkins.plugins.docker.DockerContainerWatchdog.enabled=false
I have the same issue, I'm not sure it depends on docker commands though.
This is an example of stacktrace, even though the details may differ from time to time.
The key line here is Unable to create live FilePath for jenkins-yil83x; jenkins-yil83x was marked offline: Connection was broken and I think this is a network issue.
hudson.remoting.ProxyException: java.io.StreamCorruptedException: invalid stream header: 636F7272
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:958)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:392)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
at hudson.remoting.Command.readFrom(Command.java:141)
at hudson.remoting.Command.readFrom(Command.java:127)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)
Also: hudson.remoting.ProxyException: hudson.remoting.Channel$CallSiteStackTrace: Remote call to jenkins-yil83x
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
at hudson.remoting.Request.call(Request.java:204)
at hudson.remoting.Channel.call(Channel.java:1107)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1136)
at hudson.Launcher$ProcStarter.start(Launcher.java:507)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:180)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:134)
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:330)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:323)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1295)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1184)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:118)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:114)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at jdk.internal.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:108)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at jdk.internal.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.CastBlock$ContinuationImpl.cast(CastBlock.java:47)
at jdk.internal.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
at jdk.internal.reflect.GeneratedMethodAccessor213.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:419)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:327)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:292)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
at java.base/java.lang.Thread.run(Thread.java:840)
Also: hudson.remoting.ProxyException: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 53dad5e0-1b4a-4cbd-8ac1-839f4bc6819c
Also: hudson.remoting.ProxyException: org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException: Unable to create live FilePath for jenkins-yil83x; jenkins-yil83x was marked offline: Connection was broken
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:188)
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:160)
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$Translator.get(ExecutorStepDynamicContext.java:153)
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:170)
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:160)
at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:95)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsBodySubContext.doGet(CpsBodySubContext.java:88)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:97)
at PluginClassLoader for credentials-binding//org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Callback.finished(BindingStep.java:247)
at PluginClassLoader for credentials-binding//org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Execution2$Callback2.finished(BindingStep.java:161)
at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onFailure$1(GeneralNonBlockingStepExecution.java:157)
at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
at PluginClassLoader for opentelemetry-api//io.opentelemetry.context.Context.lambda$wrap$1(Context.java:241)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused: hudson.remoting.ProxyException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 636F7272
at hudson.remoting.Request.abort(Request.java:358)
at hudson.remoting.Channel.terminate(Channel.java:1189)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:95)
Caused: hudson.remoting.ProxyException: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 636F7272
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.get(CpsBodyExecution.java:303)
at PluginClassLoader for folder-properties//com.mig82.folders.step.FolderPropertiesStep$Execution.run(FolderPropertiesStep.java:55)
at PluginClassLoader for folder-properties//com.mig82.folders.step.FolderPropertiesStep$Execution.run(FolderPropertiesStep.java:36)
at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:49)
at PluginClassLoader for opentelemetry-api//io.opentelemetry.context.Context.lambda$wrap$1(Context.java:241)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Finished: FAILURE
we have this problem too.
also if a parallel build (on gcloud node) succeeds, this it not picked up by the master (where Jenkins is started).
The agent is started by the preinstalled java, the build runs on a java we install on the fly for this build. So the build continues. But the agent is gone (java update by system)
we indeed have a vm with an update pending for the java openjdk (installed via apt) when we start a fresh node . i will disable 'unattended-upgrades' and see if this is the solution
our build just succeeded for the first time this week so this must have been the unattended-upgrades of the java jdk on the node
Just to follow up on my original observations, unattended upgrades kicking in while Jenkins agent connected, my solution is here (my code - I disable in the VM instance template):
https://github.com/GoogleCloudPlatform/horizon-sdv/blob/647d41cd1392b60d38647613733775e020deea03/workloads/android/pipelines/environment/cf_instance_template/cf_host_initialise.sh#L73
It may be a line or so overkill but it works and I've not suffered the issues ever since.