google-compute-engine-plugin icon indicating copy to clipboard operation
google-compute-engine-plugin copied to clipboard

Published 20 hours ago • verified publisher icon jenkinsci

Remote instance ci-centos-7-pjfhx1 not found locally, removing it

Open yurok15 opened this issue 6 years ago • 1 comments
trafficstars

Hello all. After restarting jenkins we begin experience issues with deleting agents. Jenkins kill agent after it successfully started. Here is a log messages:

INFO: Started provisioning ci-centos-7-pjfhx1 from gce-gce with 1 executors. Remaining excess workload: 0 Sep 24, 2019 9:35:09 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud lambda$getPlannedNodeFuture$0 INFO: Waiting 300000ms for node ci-centos-7-pjfhx1 to connect Sep 24, 2019 9:35:47 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log INFO: Launching instance: ci-centos-7-pjfhx1 Sep 24, 2019 9:35:47 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log INFO: bootstrap

INFO: 64621ms elapsed waiting for node ci-centos-7-pjfhx1 to connect Sep 24, 2019 9:37:15 AM hudson.model.Run execute

Sep 24, 2019 9:38:09 AM com.google.jenkins.plugins.computeengine.CleanLostNodesWork terminateInstance INFO: Remote instance ci-centos-7-pjfhx1 not found locally, removing it

INFO: Ping failed. Terminating the channel ci-centos-7-pjfhx1. java.util.concurrent.TimeoutException: Ping started at 1569318073002 hasn't completed by 1569318313003 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90)

INFO: I/O error in channel ci-centos-7-pjfhx1 java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861) at java.io.ObjectInputStream.(ObjectInputStream.java:357) at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:140) at hudson.remoting.Command.readFrom(Command.java:126) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

WARNING: Failed to terminate ci-centos-7-pjfhx1 com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found { "code" : 404, "errors" : [ { "domain" : "global", "message" : "The resource 'projects/tangome-corpit/zones/europe-north1-a/instances/ci-centos-7-pjfhx1' was not found", "reason" : "notFound" } ], "message" : "The resource 'projects/tangome-corpit/zones/europe-north1-a/instances/ci-centos-7-pjfhx1' was not found" } at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146) at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321) at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1067) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) at com.google.jenkins.plugins.computeengine.client.ComputeClient.terminateInstance(ComputeClient.java:365) at com.google.jenkins.plugins.computeengine.ComputeEngineInstance._terminate(ComputeEngineInstance.java:120) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy$1$1.run(OnceRetentionStrategy.java:128) at hudson.model.Queue._withLock(Queue.java:1379) at hudson.model.Queue.withLock(Queue.java:1256) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy$1.run(OnceRetentionStrategy.java:123) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

yurok15 avatar Sep 24 '19 13:09 yurok15

Hi @yurok15, I will need some more information to help debug this.

  • Was the instance ci-centos-7-pjfhx1 created before or after the restart?
  • If after, how you are provisioning agents (i.e. provisioning manually or being provisioned due to jobs queueing with the corresponding label)?
    • Related: How soon after restarting are these agents being provisioned?
  • What value is configured for "Node Retention Time"?
  • What value is configured for the "Usage"?
  • Is the option "Preemptible" checked?

stephenashank avatar Oct 10 '19 01:10 stephenashank