build icon indicating copy to clipboard operation
build copied to clipboard

ubuntu2004-armv7l: build failure

Open F3n67u opened this issue 3 years ago • 5 comments

Build link

https://ci.nodejs.org/job/node-test-binary-armv7l/1035/RUN_SUBSET=js,label=ubuntu2004-armv7l/console

Console Output

13:59:35 Running script  : #/bin/bash -x +e
13:59:35 
13:59:35 mkdir out/junit || true
13:59:35 tap2junit -i test.tap -o out/junit/test.xml || true
13:59:35 tap2junit -i cctest.tap -o out/junit/cctest.xml || true
13:59:35 FATAL: Unable to produce a script file
13:59:35 java.nio.channels.ClosedChannelException
13:59:35 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:238)
13:59:35 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
13:59:35 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
13:59:35 	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
13:59:35 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:179)
13:59:35 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:281)
13:59:35 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
13:59:35 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:246)
13:59:35 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:198)
13:59:35 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:700)
13:59:35 	at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:156)
13:59:35 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:324)
13:59:35 	at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:301)
13:59:35 	at hudson.remoting.Channel.send(Channel.java:766)
13:59:35 	at hudson.remoting.ProxyOutputStream$Chunk.lambda$execute$0(ProxyOutputStream.java:274)
13:59:35 	at hudson.remoting.PipeWriter$1.run(PipeWriter.java:159)
13:59:35 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
13:59:35 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
13:59:35 	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
13:59:35 	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
13:59:35 	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
13:59:35 	at hudson.remoting.CallableDecoratorList.lambda$applyDecorator$0(CallableDecoratorList.java:19)
13:59:35 	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
13:59:35 	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
13:59:35 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
13:59:35 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
13:59:35 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
13:59:35 	at java.lang.Thread.run(Thread.java:748)
13:59:35 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@3621c9b7:JNLP4-connect connection from 145.40.99.31/145.40.99.31:39646": Remote call on JNLP4-connect connection from 145.40.99.31/145.40.99.31:39646 failed. The channel is closing down or has closed down
13:59:35 	at hudson.remoting.Channel.call(Channel.java:994)
13:59:35 	at hudson.FilePath.act(FilePath.java:1165)
13:59:35 	at hudson.FilePath.act(FilePath.java:1154)
13:59:35 	at hudson.FilePath.createTextTempFile(FilePath.java:1580)
13:59:35 Caused: java.io.IOException: Failed to create a temp file on /home/iojs/build/workspace/node-test-binary-armv7l
13:59:35 	at hudson.FilePath.createTextTempFile(FilePath.java:1582)
13:59:35 	at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:201)
13:59:35 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:119)
13:59:35 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
13:59:35 	at hudson.plugins.postbuildtask.PostbuildTask.perform(PostbuildTask.java:123)
13:59:35 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
13:59:35 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806)
13:59:35 	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:755)
13:59:35 	at hudson.model.Build$BuildExecution.post2(Build.java:178)
13:59:35 	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:699)
13:59:35 	at hudson.model.Run.execute(Run.java:1913)
13:59:35 	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
13:59:35 	at hudson.model.ResourceController.execute(ResourceController.java:99)
13:59:35 	at hudson.model.Executor.run(Executor.java:432)
13:59:35 POST BUILD TASK : FAILURE
13:59:35 END OF POST BUILD TASK : 0
13:59:35 ERROR: Step ?Publish JUnit test result report? failed: no workspace for node-test-binary-armv7l/RUN_SUBSET=js,label=ubuntu2004-armv7l #1035
13:59:35 [PostBuildScript] - [INFO] Executing post build scripts.
13:59:35 ERROR: Build step failed with exception
13:59:35 java.lang.NullPointerException: no workspace from node hudson.slaves.DumbSlave[test-equinix-ubuntu2004_container-armv7l-2] which is computer hudson.slaves.SlaveComputer@2ad6001 and has channel null
13:59:35 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:113)
13:59:35 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
13:59:35 	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processBuildSteps(Processor.java:180)
13:59:35 	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processScripts(Processor.java:91)
13:59:35 	at org.jenkinsci.plugins.postbuildscript.processor.Processor.process(Processor.java:79)
13:59:35 	at org.jenkinsci.plugins.postbuildscript.processor.Processor.process(Processor.java:73)
13:59:35 	at org.jenkinsci.plugins.postbuildscript.MatrixPostBuildScript.perform(MatrixPostBuildScript.java:60)
13:59:35 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
13:59:35 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806)
13:59:35 	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:755)
13:59:35 	at hudson.model.Build$BuildExecution.post2(Build.java:178)
13:59:35 	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:699)
13:59:35 	at hudson.model.Run.execute(Run.java:1913)
13:59:35 	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
13:59:35 	at hudson.model.ResourceController.execute(ResourceController.java:99)
13:59:35 	at hudson.model.Executor.run(Executor.java:432)
13:59:35 Build step 'Execute scripts on Matrix' marked build as failure

F3n67u avatar Jun 28 '22 07:06 F3n67u

Another failure

Build link

https://ci.nodejs.org/job/node-test-commit-arm/42770/nodes=ubuntu2004-armv7l/console

Console output

18:38:44 ok 1771 parallel/test-module-wrap
18:38:44   ---
18:38:44 FATAL: command execution failed
19:42:09 java.nio.channels.ClosedChannelException
19:42:09 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:238)
19:42:09 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
19:42:09 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
19:42:09 	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:179)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:281)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:246)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:198)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:211)
19:42:09 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:793)
19:42:09 	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:340)
19:42:09 	at hudson.remoting.Channel.close(Channel.java:1501)
19:42:09 	at hudson.remoting.Channel.close(Channel.java:1454)
19:42:09 	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:894)
19:42:09 	at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:108)
19:42:09 	at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:774)
19:42:09 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
19:42:09 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
19:42:09 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
19:42:09 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
19:42:09 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
19:42:09 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
19:42:09 	at java.lang.Thread.run(Thread.java:748)
19:42:09 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4c349c24:JNLP4-connect connection from 145.40.99.31/145.40.99.31:53130": Remote call on JNLP4-connect connection from 145.40.99.31/145.40.99.31:53130 failed. The channel is closing down or has closed down
19:42:09 	at hudson.remoting.Channel.call(Channel.java:994)
19:42:09 	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
19:42:09 	at com.sun.proxy.$Proxy79.isAlive(Unknown Source)
19:42:09 	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1213)
19:42:09 	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1205)
19:42:09 	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:194)
19:42:09 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144)
19:42:09 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
19:42:09 	at org.jenkinsci.plugins.conditionalbuildstep.BuilderChain.perform(BuilderChain.java:71)
19:42:09 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner$2.run(BuildStepRunner.java:112)
19:42:09 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail.conditionalRun(BuildStepRunner.java:156)
19:42:09 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner.perform(BuildStepRunner.java:107)
19:42:09 	at org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder.perform(ConditionalBuilder.java:134)
19:42:09 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
19:42:09 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806)
19:42:09 	at hudson.model.Build$BuildExecution.build(Build.java:198)
19:42:09 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
19:42:09 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:514)
19:42:09 	at hudson.model.Run.execute(Run.java:1888)
19:42:09 	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
19:42:09 	at hudson.model.ResourceController.execute(ResourceController.java:99)
19:42:09 	at hudson.model.Executor.run(Executor.java:432)
19:42:09 FATAL: Unable to delete script file /tmp/jenkins7020994391648763611.sh
19:42:09 java.nio.channels.ClosedChannelException
19:42:09 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:238)
19:42:09 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
19:42:09 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
19:42:09 	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:179)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:281)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:246)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:198)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:211)
19:42:09 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:793)
19:42:09 	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
19:42:09 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:340)
19:42:09 	at hudson.remoting.Channel.close(Channel.java:1501)
19:42:09 	at hudson.remoting.Channel.close(Channel.java:1454)
19:42:09 	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:894)
19:42:09 	at hudson.slaves.SlaveComputer.access$100(SlaveComputer.java:108)
19:42:09 	at hudson.slaves.SlaveComputer$2.run(SlaveComputer.java:774)
19:42:09 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
19:42:09 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
19:42:09 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
19:42:09 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
19:42:09 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
19:42:09 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
19:42:09 	at java.lang.Thread.run(Thread.java:748)
19:42:09 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4c349c24:JNLP4-connect connection from 145.40.99.31/145.40.99.31:53130": Remote call on JNLP4-connect connection from 145.40.99.31/145.40.99.31:53130 failed. The channel is closing down or has closed down
19:42:09 	at hudson.remoting.Channel.call(Channel.java:994)
19:42:09 	at hudson.FilePath.act(FilePath.java:1165)
19:42:09 	at hudson.FilePath.act(FilePath.java:1154)
19:42:09 	at hudson.FilePath.delete(FilePath.java:1681)
19:42:09 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:162)
19:42:09 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
19:42:09 	at org.jenkinsci.plugins.conditionalbuildstep.BuilderChain.perform(BuilderChain.java:71)
19:42:09 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner$2.run(BuildStepRunner.java:112)
19:42:09 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail.conditionalRun(BuildStepRunner.java:156)
19:42:09 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner.perform(BuildStepRunner.java:107)
19:42:09 	at org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder.perform(ConditionalBuilder.java:134)
19:42:09 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
19:42:09 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806)
19:42:09 	at hudson.model.Build$BuildExecution.build(Build.java:198)
19:42:09 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
19:42:09 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:514)
19:42:09 	at hudson.model.Run.execute(Run.java:1888)
19:42:09 	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
19:42:09 	at hudson.model.ResourceController.execute(ResourceController.java:99)
19:42:09 	at hudson.model.Executor.run(Executor.java:432)
19:42:09 Build step 'Conditional steps (multiple)' marked build as failure
19:42:09 Performing Post build task...
19:42:09 Match found for : : True
19:42:09 Logical operation result is TRUE
19:42:09 Running script  : #/bin/bash -x
19:42:09 
19:42:09 mkdir -p out/junit
19:42:09 tap2junit -i test.tap -o out/junit/test.xml
19:42:09 tap2junit -i cctest.tap -o out/junit/cctest.xml
19:42:09 
19:42:09 Exception when executing the batch command : no workspace from node hudson.slaves.DumbSlave[test-equinix-ubuntu2004_container-armv7l-2] which is computer hudson.slaves.SlaveComputer@2ad6001 and has channel null
19:42:09 Build step 'Post build task' marked build as failure
19:42:09 ERROR: Step ?Publish JUnit test result report? failed: no workspace for node-test-commit-arm/nodes=ubuntu2004-armv7l #42770
19:42:09 Collecting metadata...
19:42:09 Metadata collection done.
19:42:09 Notifying upstream projects of job completion
19:42:09 Finished: FAILURE

F3n67u avatar Jul 16 '22 12:07 F3n67u

Another failure: https://ci.nodejs.org/job/node-test-binary-armv7l/1646/RUN_SUBSET=js,label=ubuntu2004-armv7l/console

@richardlau Could you help to take a look at this case?

F3n67u avatar Jul 26 '22 05:07 F3n67u

Another failure: https://ci.nodejs.org/job/node-test-binary-armv7l/1646/RUN_SUBSET=js,label=ubuntu2004-armv7l/console

@richardlau Could you help to take a look at this case?

It looks like this one was disconnected at the Jenkins server end. From the Jenkins server log:

2022-07-25 22:19:22.781+0000 [id=7464456]       SEVERE  o.j.r.p.i.SSLEngineFilterLayer#onRecv: [JNLP4-connect connection from 145.40.99.31/145.40.99.31:54122]
javax.crypto.AEADBadTagException: Tag mismatch!
        at com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:578)
        at com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
        at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
        at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
        at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:830)
        at javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
        at javax.crypto.Cipher.doFinal(Cipher.java:2459)
        at sun.security.ssl.CipherBox.decrypt(CipherBox.java:536)
        at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200)
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:963)
Caused: javax.net.ssl.SSLException: Tag mismatch!
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:970)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:345)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:677)
        at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
        at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
        at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:790)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2022-07-25 22:19:22.781+0000 [id=7464477]       INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#802789] : IO ID=8520 : seq#=8519 for test-equinix-ubuntu2004_container-armv7l-2 terminated: java.nio.channels.ClosedChannelException
2022-07-25 18:19:22:922 - AuditLog - node-test-binary-armv7l » js,ubuntu2004-armv7l #1646 Started by upstream project "node-test-binary-armv7l" build number 1,646, Parameters:[TEMP_BRANCH: {jenkins-node-test-commit-arm-fanned-d24207543e763709e7142fc4f228464910093386-binary-pi1p}, CONFIG_FLAGS: {}, NODES_SUBSET: {${NODES_SUBSET}}, IGNORE_FLAKY_TESTS: {true}] on node test-equinix-ubuntu2004_container-armv7l-2 started at 2022-07-25T22:18:23Z completed in 58849ms completed: FAILURE
2022-07-25 22:19:22.948+0000 [id=7460504]       INFO    o.j.p.s.g.l.RunStatsListener#onFinalized: RUN_SUBSET=js,label=ubuntu2004-armv7l build is completed its status is : FAILURE at time : Mon Jul 25 18:19:22 EDT 2022
2022-07-25 22:19:22.948+0000 [id=7460504]       INFO    o.j.p.a.FailureListener#onFinalized: Monitoring test-equinix-ubuntu2004_container-armv7l-2 after failed run of node-test-binary-armv7l » js,ubuntu2004-armv7l #1646
2022-07-25 22:19:22.948+0000 [id=7460504]       WARNING o.j.p.a.FailureListener#onFinalized: test-equinix-ubuntu2004_container-armv7l-2 taken offline after failed run of node-test-binary-armv7l » js,ubuntu2004-armv7l #1646

richardlau avatar Jul 26 '22 12:07 richardlau

I think we're seen javax.crypto.AEADBadTagException: Tag mismatch! elsewhere, e.g. https://ci.nodejs.org/job/node-test-commit-arm/42954/nodes=ubuntu2004-armv7l/console

08:07:16 FATAL: command execution failed
08:07:16 javax.crypto.AEADBadTagException: Tag mismatch!
08:07:16 	at com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:578)
08:07:16 	at com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
08:07:16 	at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
08:07:16 	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
08:07:16 	at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
08:07:16 	at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:830)
08:07:16 	at javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
08:07:16 	at javax.crypto.Cipher.doFinal(Cipher.java:2459)
08:07:16 	at sun.security.ssl.CipherBox.decrypt(CipherBox.java:536)
08:07:16 	at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200)
08:07:16 	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:963)
08:07:16 Caused: javax.net.ssl.SSLException: Tag mismatch!
08:07:16 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
08:07:16 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
08:07:16 	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:970)
08:07:16 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
08:07:16 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
08:07:16 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
08:07:16 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:345)
08:07:16 	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
08:07:16 	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:677)
08:07:16 	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
08:07:16 	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
08:07:16 	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:790)
08:07:16 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
08:07:16 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
08:07:16 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
08:07:16 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
08:07:16 	at java.lang.Thread.run(Thread.java:748)
08:07:16 Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from 145.40.99.31/145.40.99.31:58280' is disconnected.
08:07:16 	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:216)
08:07:16 	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
08:07:16 	at com.sun.proxy.$Proxy79.isAlive(Unknown Source)
08:07:16 	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1213)
08:07:16 	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1205)
08:07:16 	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:194)
08:07:16 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144)
08:07:16 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
08:07:16 	at org.jenkinsci.plugins.conditionalbuildstep.BuilderChain.perform(BuilderChain.java:71)
08:07:16 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner$2.run(BuildStepRunner.java:112)
08:07:16 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail.conditionalRun(BuildStepRunner.java:156)
08:07:16 	at org.jenkins_ci.plugins.run_condition.BuildStepRunner.perform(BuildStepRunner.java:107)
08:07:16 	at org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder.perform(ConditionalBuilder.java:134)
08:07:16 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
08:07:16 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806)
08:07:16 	at hudson.model.Build$BuildExecution.build(Build.java:198)
08:07:16 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
08:07:16 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:514)
08:07:16 	at hudson.model.Run.execute(Run.java:1888)
08:07:16 	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
08:07:16 	at hudson.model.ResourceController.execute(ResourceController.java:99)
08:07:16 	at hudson.model.Executor.run(Executor.java:432)

which was on the Jenkins agent side (while the one in the previous comment had the exception on the Jenkins server side). I'm not sure if we've seen this outside of armv7l.

richardlau avatar Jul 26 '22 12:07 richardlau

The host for the test-equinix-ubuntu2004_container-armv7l-2 container had 65 zombie processes

[rlau@drx-hemera build]$ ssh test-equinix-ubuntu2004_docker-arm64-3
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-92-generic aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Tue Jul 26 11:40:58 UTC 2022

  System load:              0.11
  Usage of /:               21.6% of 870.81GB
  Memory usage:             8%
  Swap usage:               0%
  Processes:                1625
  Users logged in:          0
  IPv4 address for bond0:   145.40.99.31
  IPv6 address for bond0:   2604:1380:4111:eb00::7
  IPv4 address for docker0: 172.17.0.1

  => There are 65 zombie processes.

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

13 updates can be applied immediately.
To see these additional updates run: apt list --upgradable


*** System restart required ***
Last login: Fri Jul 22 12:58:35 2022 from 80.2.248.72
root@test-equinix-ubuntu2004-docker-arm64-3:~#

which were all linked to the armv7l-2 container:

root@test-equinix-ubuntu2004-docker-arm64-3:~# ps -ef | grep 497267
iojs       61169  497267  0 Jul09 ?        00:00:00 [bash] <defunct>
iojs      141279  497267  0 Jul13 ?        00:00:00 [bash] <defunct>
iojs      141715  497267  0 Jul10 ?        00:00:00 [bash] <defunct>
iojs      199135  497267  0 Jul08 ?        00:00:00 [bash] <defunct>
iojs      272541  497267  0 Jul18 ?        00:00:00 [bash] <defunct>
iojs      305459  497267  0 Jun30 ?        00:00:00 [bash] <defunct>
iojs      382146  497267  0 Jun21 ?        00:00:00 [bash] <defunct>
iojs      397570  497267  0 Jul24 ?        00:00:00 [bash] <defunct>
iojs      417266  497267  0 Jul01 ?        00:00:00 [bash] <defunct>
iojs      422371  497267  0 Jul24 ?        00:00:00 [bash] <defunct>
iojs      426205  497267  0 Jul11 ?        00:00:00 [bash] <defunct>
iojs      497267  497246  0 Jun20 ?        06:00:31 java -Xmx128m -jar /home/iojs/slave.jar -jnlpUrl https://ci.nodejs.org/computer/test-equinix-ubuntu2004_container-armv7l-2/slave-agent.jnlp -secret

I've restarted the container. This has cleared up the zombie processes. I don't know if it'll address the AEADBadTagException exceptions.

richardlau avatar Jul 26 '22 12:07 richardlau

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

github-actions[bot] avatar May 23 '23 00:05 github-actions[bot]