byte-buddy icon indicating copy to clipboard operation
byte-buddy copied to clipboard

Published 20 hours ago • verified publisher icon raphw

Occasional deadlock occurs when using bytebuddy when tomcat starts

Open luffy1849 opened this issue 2 years ago • 9 comments

This seems to be caused by tomcat's listener JreMemoryLeakPreventionListener

I also found other people on the Internet who had the same problem as me https://discuss.elastic.co/t/use-javaagent-happened-deadlock/320132

Found one Java-level deadlock:
=============================
"main":
  waiting to lock monitor 0x00007ffa6c07c238 (object 0x0000000733346268, a sun.net.www.protocol.jar.JarFileFactory),
  which is held by "org.apache.skywalking.apm.dependencies.kafka-admin-client-thread | adminclient-1"
"org.apache.skywalking.apm.dependencies.kafka-admin-client-thread | adminclient-1":
  waiting to lock monitor 0x00007ffa6c07cff8 (object 0x0000000733346218, a sun.net.www.protocol.file.Handler),
  which is held by "main"

Java stack information for the threads listed above:
===================================================
"main":
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:80)
	- waiting to lock <0x0000000733346268> (a sun.net.www.protocol.jar.JarFileFactory)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:453)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:434)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$Compound.locate(ClassFileLocator.java:1891)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:839)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:921)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1019)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1088)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getSuperClass(TypeDescription.java:8358)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfNonGenericType.getSuperClass(TypeDescription.java:3602)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDefinition$SuperClassIterator.next(TypeDefinition.java:391)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDefinition$SuperClassIterator.next(TypeDefinition.java:357)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:54)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:32)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.agent.core.plugin.match.ProtectiveShieldMatcher.matches(ProtectiveShieldMatcher.java:47)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1833)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11838)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11555)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12178)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11747)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at java.net.URL.getURLStreamHandler(URL.java:1175)
	at java.net.URL.<init>(URL.java:420)
	at java.net.URL.<init>(URL.java:310)
	at java.net.URL.<init>(URL.java:333)
	at sun.net.www.protocol.file.Handler.openConnection(Handler.java:92)
	- locked <0x0000000733346218> (a sun.net.www.protocol.file.Handler)
	at sun.net.www.protocol.file.Handler.openConnection(Handler.java:72)
	- locked <0x0000000733346218> (a sun.net.www.protocol.file.Handler)
	at java.net.URL.openConnection(URL.java:979)
	at sun.net.www.protocol.jar.JarURLConnection.<init>(JarURLConnection.java:84)
	at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:41)
	at java.net.URL.openConnection(URL.java:979)
	at org.apache.catalina.core.JreMemoryLeakPreventionListener.lifecycleEvent(JreMemoryLeakPreventionListener.java:413)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
	- locked <0x000000073338c368> (a org.apache.catalina.core.StandardServer)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:101)
	- locked <0x000000073338c368> (a org.apache.catalina.core.StandardServer)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:581)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:604)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:310)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)
"org.apache.skywalking.apm.dependencies.kafka-admin-client-thread | adminclient-1":
	at sun.net.www.protocol.file.Handler.openConnection(Handler.java:72)
	- waiting to lock <0x0000000733346218> (a sun.net.www.protocol.file.Handler)
	at java.net.URL.openConnection(URL.java:979)
	at sun.net.www.protocol.jar.JarFileFactory.getConnection(JarFileFactory.java:65)
	at sun.net.www.protocol.jar.JarFileFactory.getPermission(JarFileFactory.java:154)
	at sun.net.www.protocol.jar.JarFileFactory.getCachedJarFile(JarFileFactory.java:126)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:81)
	- locked <0x0000000733346268> (a sun.net.www.protocol.jar.JarFileFactory)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:453)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:434)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$Compound.locate(ClassFileLocator.java:1891)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:839)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:921)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1019)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1088)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getEnclosingType(TypeDescription.java:8403)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$ForParameterizedType$LazyParameterizedType.getOwnerType(TypePool.java:5409)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$Visitor$Substitutor.onParameterizedType(TypeDescription.java:1876)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$Visitor$Substitutor$WithoutTypeSubstitution.onParameterizedType(TypeDescription.java:1925)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfParameterizedType.accept(TypeDescription.java:5057)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$LazyProjection.accept(TypeDescription.java:6220)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$LazyProjection$WithResolvedErasure.resolve(TypeDescription.java:6872)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$LazyProjection$WithEagerNavigation.getSuperClass(TypeDescription.java:6495)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDefinition$SuperClassIterator.next(TypeDefinition.java:391)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDefinition$SuperClassIterator.next(TypeDefinition.java:357)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:54)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:32)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.agent.core.plugin.match.ProtectiveShieldMatcher.matches(ProtectiveShieldMatcher.java:47)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1833)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11838)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11555)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12178)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11747)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at javax.security.auth.SubjectDomainCombiner.<init>(SubjectDomainCombiner.java:52)
	at javax.security.auth.Subject$2.run(Subject.java:563)
	at javax.security.auth.Subject$2.run(Subject.java:558)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.createContext(Subject.java:558)
	at javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:179)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:171)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:252)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:200)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.SaslChannelBuilder$$Lambda$47/213974480.get(Unknown Source)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:140)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:208)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:336)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.Selector.registerChannel(Selector.java:327)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.network.Selector.connect(Selector.java:259)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:951)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:291)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:969)
	at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1184)
	at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.

Other similar deadlock details


Found one Java-level deadlock:
=============================
"pool-5-thread-1":
  waiting to lock monitor 0x00007f93f800e288 (object 0x0000000733344d68, a sun.net.www.protocol.file.Handler),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007f9434003ab8 (object 0x00000007333442d8, a sun.net.www.protocol.jar.JarFileFactory),
  which is held by "pool-5-thread-1"

Java stack information for the threads listed above:
===================================================
"pool-5-thread-1":
	at sun.net.www.protocol.file.Handler.openConnection(Handler.java:72)
	- waiting to lock <0x0000000733344d68> (a sun.net.www.protocol.file.Handler)
	at java.net.URL.openConnection(URL.java:979)
	at sun.net.www.protocol.jar.JarFileFactory.getConnection(JarFileFactory.java:65)
	at sun.net.www.protocol.jar.JarFileFactory.getPermission(JarFileFactory.java:154)
	at sun.net.www.protocol.jar.JarFileFactory.getCachedJarFile(JarFileFactory.java:126)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:81)
	- locked <0x00000007333442d8> (a sun.net.www.protocol.jar.JarFileFactory)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:453)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:434)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$Compound.locate(ClassFileLocator.java:1891)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:839)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:921)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1019)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1088)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getInterfaces(TypeDescription.java:8365)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfParameterizedType.getInterfaces(TypeDescription.java:4927)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$LazyProjection$WithEagerNavigation.getInterfaces(TypeDescription.java:6502)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:67)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:32)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.agent.core.plugin.match.ProtectiveShieldMatcher.matches(ProtectiveShieldMatcher.java:47)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1833)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11838)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11555)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12178)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11747)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at java.util.concurrent.ConcurrentHashMap$EntryIterator.next(ConcurrentHashMap.java:3461)
	at java.util.concurrent.ConcurrentHashMap$EntryIterator.next(ConcurrentHashMap.java:3446)
	at com.dsfsd.api.vmmonitor.metrics.TimedLogTask.run(TimedLogTask.java:71)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"main":
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:80)
	- waiting to lock <0x00000007333442d8> (a sun.net.www.protocol.jar.JarFileFactory)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:453)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:434)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.ClassFileLocator$Compound.locate(ClassFileLocator.java:1891)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:839)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:921)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1019)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1088)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getSuperClass(TypeDescription.java:8358)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfNonGenericType.getSuperClass(TypeDescription.java:3602)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDefinition$SuperClassIterator.next(TypeDefinition.java:391)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDefinition$SuperClassIterator.next(TypeDefinition.java:357)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:54)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:32)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.agent.core.plugin.match.ProtectiveShieldMatcher.matches(ProtectiveShieldMatcher.java:47)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1833)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11838)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11555)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12238)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12178)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11747)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at java.net.URL.getURLStreamHandler(URL.java:1175)
	at java.net.URL.<init>(URL.java:420)
	at java.net.URL.<init>(URL.java:310)
	at java.net.URL.<init>(URL.java:333)
	at sun.net.www.protocol.file.Handler.openConnection(Handler.java:92)
	- locked <0x0000000733344d68> (a sun.net.www.protocol.file.Handler)
	at sun.net.www.protocol.file.Handler.openConnection(Handler.java:72)
	- locked <0x0000000733344d68> (a sun.net.www.protocol.file.Handler)
	at java.net.URL.openConnection(URL.java:979)
	at sun.net.www.protocol.jar.JarURLConnection.<init>(JarURLConnection.java:84)
	at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:41)
	at java.net.URL.openConnection(URL.java:979)
	at org.apache.catalina.core.JreMemoryLeakPreventionListener.lifecycleEvent(JreMemoryLeakPreventionListener.java:413)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
	- locked <0x00000007333a2908> (a org.apache.catalina.core.StandardServer)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:101)
	- locked <0x00000007333a2908> (a org.apache.catalina.core.StandardServer)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:581)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:604)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:310)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)

Found 1 deadlock.

luffy1849 avatar Apr 25 '23 03:04 luffy1849

The lock occurs as two threads attempt to load a class which requires a lock for looking up a class file. The same issue can however occur during regular class loading. I would not know that this is related to class loading and is likely a bug. Are you running an outdated version of Tomcat?

raphw avatar Apr 25 '23 19:04 raphw

The tomcat version is not out of date And I'm not sure if this problem can be avoided by excluding bytebuddy from loading some class

using catalina jvm flags: -Dsun.net.spi.nameservice.provider.1=dns,dnsjava -Dsun.net.spi.nameservice.provider.2=default  -Xmx6g -Xms6g -Xmn3g -Xss1m -javaagent:/data1/insight-agent/insight-agent.jar -Xss1m -XX:MaxTenuringThreshold=4 -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=8 -XX:CMSInitiatingOccupancyFraction=70 -XX:+ExplicitGCInvokesConcurrent -server -DServer=live-stream-dispatcher -XX:-OmitStackTraceInFastThrow -XX:+PrintFlagsFinal -XX:+PrintCommandLineFlags -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:../gclogs/gc.log.20230426_101614 -XX:CICompilerCount=6
Using CATALINA_BASE:   /data1
Using CATALINA_HOME:   /data1
Using CATALINA_TMPDIR: /data1/temp
Using JRE_HOME:        /usr/jdk1.8.0_121
Using CLASSPATH:       /data1/bin/bootstrap.jar:/data1/bin/tomcat-juli.jar
Server version: Apache Tomcat/8.0.46
Server built:   Aug 10 2017 10:10:31 UTC
Server number:  8.0.46.0
OS Name:        Linux
OS Version:     4.19.91-21.2.al7.x86_64
Architecture:   amd64
JVM Version:    1.8.0_121-b13
JVM Vendor:     Oracle Corporation

luffy1849 avatar Apr 26 '23 02:04 luffy1849

Byte Buddy is not loading a class, it is looking up a class file. This happens during instrumentation, I am not really sure how this can trigger a deadlock for Tomcat.

raphw avatar Apr 26 '23 21:04 raphw

Facing the same on Server version: Apache Tomcat/9.0.58. It does not happen everytime though.

amar1019 avatar Jun 02 '23 05:06 amar1019

Seems like it is happening when agent's own classes are getting loaded. It is not consistent. Resolved it by excluding all agent's classes. Earlier i was excluding only bytebuddy package.

amar1019 avatar Jun 02 '23 10:06 amar1019

Seems like it is happening when agent's own classes are getting loaded. It is not consistent. Resolved it by excluding all agent's classes. Earlier i was excluding only bytebuddy package.

I haven't solved this yet. I don't quite understand. For example, which classes are excluded?

luffy1849 avatar Jun 02 '23 10:06 luffy1849

In my case, i am excluding all the agents' classes including bytebuddy( com.test., which includes com.test.bytebuddy. and com.test.advice.* and com.test.util.). Earlier i was giving com.test.bytebuddy.

amar1019 avatar Jun 06 '23 04:06 amar1019

You should not instrument the agent from within its own class loader, which should factor out the agent.

Ideally, you bundle the agent in a custom jar and when loading the agent, you create a new class loader which then loads the agent within its own class loader. From this agent, exclude this dedicated class loader.

raphw avatar Jun 06 '23 09:06 raphw

image

I solved this problem by excluding sun.net.www.protocol

luffy1849 avatar Jun 25 '23 10:06 luffy1849