azure-vm-agents-plugin icon indicating copy to clipboard operation
azure-vm-agents-plugin copied to clipboard

Published 20 hours ago • verified publisher icon jenkinsci

VM not deleted if failure occurs in #terminateVirtualMachine

Open henti opened this issue 4 years ago • 2 comments

Version report

Jenkins and plugins versions report:

Jenkins : 
Debian 8 64bit
Openjdk version "1.8.0_171"
Jenkins 2.263.4 
Metrics Plugin (metrics): 4.0.2.7
PAM Authentication plugin (pam-auth): 1.6
Pipeline: Model API (pipeline-model-api): 1.7.2
Run Condition Plugin (run-condition): 1.5
Pipeline Utility Steps (pipeline-utility-steps): 2.6.1
Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 1.0
ECharts API Plugin (echarts-api): 4.9.0-2
Popper.js API Plugin (popper-api): 1.16.0-7
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 2.1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4
JQuery3 API Plugin (jquery3-api): 3.5.1-2
jQuery plugin (jquery): 1.12.4-1
Authentication Tokens API Plugin (authentication-tokens): 1.4
Credentials Binding Plugin (credentials-binding): 1.24
Icon Shim Plugin (icon-shim): 2.0.3
Pipeline: Input Step (pipeline-input-step): 2.12
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
GitHub Branch Source Plugin (github-branch-source): 2.9.3
Command Agent Launcher Plugin (command-launcher): 1.5
Plugin Utilities API Plugin (plugin-util-api): 1.6.1
build log file size checker plugin (logfilesizechecker): 1.5
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Matrix Project Plugin (matrix-project): 1.18
Resource Disposer Plugin (resource-disposer): 0.14
Build Timeout (build-timeout): 1.20
Pipeline: Declarative (pipeline-model-definition): 1.7.2
Jackson 2 API Plugin (jackson2-api): 2.12.0
Trilead API Plugin (trilead-api): 1.0.13
Copy Artifact Plugin (copyartifact): 1.45.3
SCM API Plugin (scm-api): 2.6.4
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.7.2
Priority Sorter Plugin (PrioritySorter): 3.6.0
Pipeline: Stage Step (pipeline-stage-step): 2.5
Azure Commons Plugin (azure-commons): 1.0.5
Email Extension Plugin (email-ext): 2.80
Pipeline: Multibranch (workflow-multibranch): 2.22
Checks API plugin (checks-api): 1.2.0
Gradle Plugin (gradle): 1.36
OkHttp Plugin (okhttp-api): 3.14.9
Pipeline: Groovy (workflow-cps): 2.87
Pipeline (workflow-aggregator): 2.6
Pipeline: API (workflow-api): 2.40
Script Security Plugin (script-security): 1.75
Folders Plugin (cloudbees-folder): 6.15
Build Failure Analyzer (build-failure-analyzer): 1.27.1
MSBuild Plugin (msbuild): 1.29
GIT server Plugin (git-server): 1.9
SSH Agent Plugin (ssh-agent): 1.20
Branch API Plugin (branch-api): 2.6.2
promoted builds plugin (promoted-builds): 3.7
Matrix Authorization Strategy Plugin (matrix-auth): 2.6.4
Pipeline: Stage View Plugin (pipeline-stage-view): 2.19
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.17
SSH Credentials Plugin (ssh-credentials): 1.18.1
SSH Build Agents plugin (ssh-slaves): 1.31.5
Timestamper (timestamper): 1.11.8
Job DSL (job-dsl): 1.77
Ant Plugin (ant): 1.11
Pipeline: SCM Step (workflow-scm-step): 2.11
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10
Bootstrap 4 API Plugin (bootstrap4-api): 4.5.3-1
External Monitor Job Type Plugin (external-monitor-job): 1.7
Mailer Plugin (mailer): 1.32.1
GitHub API Plugin (github-api): 1.117
Azure Credentials (azure-credentials): 4.0.5
WMI Windows Agents Plugin (windows-slaves): 1.7
Throttle Concurrent Builds Plug-in (throttle-concurrents): 2.1
Font Awesome API Plugin (font-awesome-api): 5.15.1-1
Conditional BuildStep (conditional-buildstep): 1.4.1
JSch dependency plugin (jsch): 0.1.55.2
Node and Label parameter plugin (nodelabelparameter): 1.7.2
Maven Integration plugin (maven-plugin): 3.8
Pipeline: Basic Steps (workflow-basic-steps): 2.22
Token Macro Plugin (token-macro): 2.13
Snakeyaml API Plugin (snakeyaml-api): 1.27.0
Rebuilder (rebuild): 1.31
Workspace Cleanup Plugin (ws-cleanup): 0.38
jQuery UI plugin (jquery-ui): 1.0.2
Pipeline: REST API Plugin (pipeline-rest-api): 2.19
Async Http Client (async-http-client): 1.9.40.0
Pipeline: Step API (workflow-step-api): 2.23
Credentials Plugin (credentials): 2.3.14
Structs Plugin (structs): 1.20
LDAP Plugin (ldap): 1.26
Git client plugin (git-client): 3.6.0
Docker Pipeline (docker-workflow): 1.25
Plain Credentials Plugin (plain-credentials): 1.7
MapDB API Plugin (mapdb-api): 1.0.9.0
Javadoc Plugin (javadoc): 1.6
Variant Plugin (variant): 1.4
GitHub Organization Folder Plugin (github-organization-folder): 1.6
GitHub Pull Request Builder (ghprb): 1.42.1
GitHub Authentication plugin (github-oauth): 0.33
Azure VM Agents (azure-vm-agents): 1.5.1
GitHub plugin (github): 1.32.0
Display URL API (display-url-api): 2.3.4
Subversion Plug-in (subversion): 2.13.2
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Parameterized Trigger plugin (parameterized-trigger): 2.39
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.37
Pipeline: Build Step (pipeline-build-step): 2.13
Pipeline: Supporting APIs (workflow-support): 3.7
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
Groovy (groovy): 2.3
Job Configuration History Plugin (jobConfigHistory): 2.26
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Durable Task Plugin (durable-task): 1.35
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.7.2
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.13-1.0
JUnit Plugin (junit): 1.48
Docker Commons Plugin (docker-commons): 1.17
Build Alias Setter Plugin (build-alias-setter): 0.4
Git plugin (git): 4.5.2
Pipeline: Job (workflow-job): 2.40
bouncycastle API Plugin (bouncycastle-api): 2.18
Cloud Statistics Plugin (cloud-stats): 0.25
Lockable Resources plugin (lockable-resources): 2.10

Agent :
Debian Buster 64bit
openjdk version "1.8.0_265"

Reproduction steps

  • Run plugin with template
  • VM is idle
  • Agent shuts down VM

Results

Expected result:

  • VM is shut down

Actual result:

  • VM shutdown fails
  • VM is not deleted.

Logs from Jenkins.

2021-04-01 21:54:37.158+0000 [id=812700]        INFO    c.m.azure.vmagent.AzureVMAgent#deprovision: AzureVMAgent: deprovision: Deprovision called for agent jenkins-8core-workerf73650, for reason: Node is being deleted by Jenkins after idle timeout
2021-04-01 21:54:37.630+0000 [id=811938]        INFO    o.j.p.workflow.job.WorkflowRun#finish: ODASA #19127 completed: SUCCESS
2021-04-01 21:54:57.174+0000 [id=812700]        INFO    c.m.azure.vmagent.AzureVMAgent#deprovision: AzureVMAgent: deprovision: Template terminate script,
2021-04-01 21:54:57.174+0000 [id=812700]        INFO    c.m.azure.vmagent.AzureVMAgent#deprovision: AzureVMAgent: deprovision: skipping terminate script execution.
2021-04-01 21:54:57.174+0000 [id=812700]        INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for jenkins-8core-workerf73650
2021-04-01 21:54:57.175+0000 [id=812700]        WARNING c.m.a.v.AzureVMManagementServiceDelegate#terminateVirtualMachine: AzureVMManagementServiceDelegate: terminateVirtualMachine: while deleting VM
java.net.UnknownHostException: management.azure.com
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at okhttp3.Dns$1.lookup(Dns.java:40)
        at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
        at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:149)
        at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84)
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:214)
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:54)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.vmagent.AzureVMAgentPlugin$AzureTelemetryInterceptor.intercept(AzureVMAgentPlugin.java:29)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254)
        at okhttp3.RealCall.execute(RealCall.java:92)
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:186)
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40)
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
        at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
        at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.subscribe(Observable.java:10423)
        at rx.Observable.subscribe(Observable.java:10390)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443)
Caused: java.lang.RuntimeException
        at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
        at rx.observables.BlockingObservable.last(BlockingObservable.java:226)
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:110)
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:34)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.virtualMachineExists(AzureVMManagementServiceDelegate.java:930)
Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException
        at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54)
        at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.virtualMachineExists(AzureVMManagementServiceDelegate.java:932)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.terminateVirtualMachine(AzureVMManagementServiceDelegate.java:1728)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.terminateVirtualMachine(AzureVMManagementServiceDelegate.java:1698)
        at com.microsoft.azure.vmagent.AzureVMAgent.deprovision(AzureVMAgent.java:654)
        at com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy$1.call(AzureVMCloudRetensionStrategy.java:102)
        at com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy$1.call(AzureVMCloudRetensionStrategy.java:91)
        at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

henti avatar Apr 06 '21 10:04 henti

What makes this an issue with this plugin?

It appears to be an issue with your DNS, does the plugin later retry deleting the VM?

timja avatar Apr 06 '21 10:04 timja

That is the bug,

The VM is kept in "off-line" mode in Jenkins but never deleted. The VM is not shutdown and the plugin does not attempt to delete the VM at a later stage.

henti avatar Apr 06 '21 12:04 henti

I wasn't able to reproduce this, maybe its been improved by https://github.com/jenkinsci/jenkins/pull/9855

I tried with both once and idle and they retried just fine

timja avatar Dec 20 '24 16:12 timja