kubernetes-client
kubernetes-client copied to clipboard
Published
20 hours ago •
fabric8io
fabric8io
Deadlock when there is a wrongly configured client
Describe the bug
Deadlock seems to happen when a jenkins kubernetes agent is trying to be created but it cannot due to some error. In this case having wrongly configured a container.
Fabric8 Kubernetes Client version
6.4.1
Steps to reproduce
Example:
podTemplate(containers: [
containerTemplate(name: 'maven', image: 'wrong_name:3.0.0', command: 'sleep', args: '99'),
]) {
retry(count: 100, conditions: [kubernetesAgent(), nonresumable()]) {
node(POD_LABEL) {
stage('Get a Maven project') {
container('maven') {
stage('Hello')
{ sh 'echo Hello' }
}
}
}
}
}
"wrong_name" image does not exists and then it retries 100 times.
While this pod is being created, when going to manage/configure and pushing save button; configuration is not saved and an error 504 appears in browser.
Performing a thread dump, this thread appears not to finish:
Handling POST /manage/configSubmit from 10.133.236.30 : https-jsse-nio-8443-exec-10
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
[email protected]/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
[email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
[email protected]/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
[email protected]/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:162)
io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:185)
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:611)
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:598)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchCloud(Reaper.java:221)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchClouds(Reaper.java:194)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$ReaperSaveableListener.onChange(Reaper.java:532)
hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:82)
jenkins.model.Jenkins.save(Jenkins.java:3591)
hudson.BulkChange.commit(BulkChange.java:98)
jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:4036)
[email protected]/java.lang.invoke.LambdaForm$DMH/0x00000008004c6c40.invokeVirtual(LambdaForm$DMH)
[email protected]/java.lang.invoke.LambdaForm$MH/0x00000008004c3c40.invoke(LambdaForm$MH)
[email protected]/java.lang.invoke.LambdaForm$MH/0x00000008004c5040.invoke(LambdaForm$MH)
[email protected]/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
[email protected]/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)
org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48)
org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26)
org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
org.kohsuke.stapler.Stapler.service(Stapler.java:248)
javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
[email protected]/java.lang.Thread.run(Thread.java:829)
And blocks other threads:
"Handling GET /tcpSlaveAgentListener/ from 10.133.236.30 : https-jsse-nio-8443-exec-2 TcpSlaveAgentListener/index.jelly" Id=36 Group=main BLOCKED on hudson.model.Hudson@42d24189 owned by "Handling POST /manage/configSubmit from 10.133.236.30 : https-jsse-nio-8443-exec-10" Id=44
at jenkins.model.Jenkins.getAgentProtocols(Jenkins.java:1255)
- blocked on hudson.model.Hudson@42d24189
at hudson.TcpSlaveAgentListener.getAgentProtocolNames(TcpSlaveAgentListener.java:163)
at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76)
at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:259)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:284)
at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:104)
at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:172)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@44b1df47
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at [email protected]/java.lang.Thread.run(Thread.java:829)
Number of locked synchronizers = 1
- org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker@6e02fcd8
Checking the calls it seems to be a problem happening Kubernetes client code in https://github.com/fabric8io/kubernetes-client/blob/v6.8.1/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/BaseOperation.java#L611 introduced by https://github.com/fabric8io/kubernetes-client/commit/0ebd164432cc2e1e6856a49ed82d3edcf3b84564
THis problem started appearing after change in Jenkins core: https://github.com/jenkinsci/jenkins/blame/jenkins-2.414.2/core/src/main/java/jenkins/model/Jenkins.java#L1254 introduced by https://github.com/jenkinsci/jenkins/commit/81bf589596acdd54e3dafb9a7d4cdb66a0029034
Expected behavior
No deadlock should appear.
Runtime
Kubernetes (vanilla)
Kubernetes API Server version
1.24
Environment
Linux
Fabric8 Kubernetes Client Logs
No response
Additional context
No response
@AndresSanchezG is it a deadlock, or just that one of your theads is holding a lock that is needed by another? A deadlock isn't clear from what you provided.
Also the lock involved "blocked on hudson.model.Hudson@42d24189" isn't coming from fabric8. Since fabric8 can make blocking calls you may want to reconsider how you are holding that lock.
Are you expecting the watch call to fail fast? That may have to do with the number of retries that are allowed - if the misconfiguration is producing an error that seems recoverable, then we may keep trying the operation.
@shawkins, so from the threads and traces it seems that while a pod is trying to start up the lock is not released. May be not a deadlock but there is a situation where the lock is never released and therefore it impacts the system. I have already spoken with Jenkins core developers and it seems to be on the part of the plugin.
This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}