Disable instrumentation for external libraries

[jarasez]

We are using java.lang.GroovyClassLoader, which has a dependency to a cache org.codehaus.groovy.runtime.memoize.StampedCommonCache (groovy version 3.0.10). When instrumentation is added, it results in a deadlock. Tried using dd.trace.classes.exclude=groovy.lang.GroovyClassLoader (and other combinations), but with no success. Stacktrace: java.lang.Thread.State: WAITING at jdk.internal.misc.Unsafe.park(Unsafe.java:-1) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1315) at java.util.concurrent.locks.StampedLock.writeLock(StampedLock.java:461) at org.codehaus.groovy.runtime.memoize.StampedCommonCache.doWithWriteLock(StampedCommonCache.java:270) at org.codehaus.groovy.runtime.memoize.StampedCommonCache.remove(StampedCommonCache.java:222) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:950) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:910) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:979) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:967) at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:607) at datadog.trace.agent.tooling.bytebuddy.DDCachingPoolStrategy$CachingResolutionForMaybeLoadableType.resolve(DDCachingPoolStrategy.java:311) at net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1088) at net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getModifiers(TypeDescription.java:8444) at net.bytebuddy.description.ModifierReviewable$AbstractBase.matchesMask(ModifierReviewable.java:618) at net.bytebuddy.description.ModifierReviewable$AbstractBase.isPublic(ModifierReviewable.java:336) at net.bytebuddy.description.type.TypeDescription$AbstractBase.isVisibleTo(TypeDescription.java:7901) at net.bytebuddy.matcher.VisibilityMatcher.doMatch(VisibilityMatcher.java:48) at net.bytebuddy.matcher.VisibilityMatcher.doMatch(VisibilityMatcher.java:27) at net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249) at net.bytebuddy.matcher.NegatingMatcher.matches(NegatingMatcher.java:47) at net.bytebuddy.matcher.ErasureMatcher.doMatch(ErasureMatcher.java:50) at net.bytebuddy.matcher.ErasureMatcher.doMatch(ErasureMatcher.java:29) at net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249) at net.bytebuddy.matcher.MethodParameterTypeMatcher.doMatch(MethodParameterTypeMatcher.java:48) at net.bytebuddy.matcher.MethodParameterTypeMatcher.doMatch(MethodParameterTypeMatcher.java:27) at net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249) at net.bytebuddy.matcher.CollectionItemMatcher.doMatch(CollectionItemMatcher.java:48) at net.bytebuddy.matcher.CollectionItemMatcher.doMatch(CollectionItemMatcher.java:26) at net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249) at net.bytebuddy.matcher.NegatingMatcher.matches(NegatingMatcher.java:47) at net.bytebuddy.matcher.MethodParametersMatcher.doMatch(MethodParametersMatcher.java:49) at net.bytebuddy.matcher.MethodParametersMatcher.doMatch(MethodParametersMatcher.java:28) at net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249) at net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146) at net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:482) at net.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:204) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11763) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11698) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11415) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12178) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12110) at java.security.AccessController.doPrivileged(AccessController.java:-1) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java:11593) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11641) at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source:-1) at datadog.trace.agent.tooling.bytebuddy.DDJava9ClassFileTransformer.transform(DDJava9ClassFileTransformer.java:62) at sun.instrument.TransformerManager.transform(TransformerManager.java:188) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563) at java.lang.ClassLoader.defineClass1(ClassLoader.java:-1) at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at groovy.lang.GroovyClassLoader.access$400(GroovyClassLoader.java:90) at groovy.lang.GroovyClassLoader$ClassCollector.createClass(GroovyClassLoader.java:700) at groovy.lang.GroovyClassLoader$ClassCollector.onClassNode(GroovyClassLoader.java:717) at groovy.lang.GroovyClassLoader$ClassCollector.call(GroovyClassLoader.java:721) at org.codehaus.groovy.control.CompilationUnit$3.lambda$call$0(CompilationUnit.java:806) at org.codehaus.groovy.control.CompilationUnit$3$$Lambda$1867.439417963.accept(Unknown Source:-1) at java.util.Optional.ifPresent(Optional.java:183) at org.codehaus.groovy.control.CompilationUnit$3.call(CompilationUnit.java:806) at org.codehaus.groovy.control.CompilationUnit$3.call(CompilationUnit.java:813) at org.codehaus.groovy.control.CompilationUnit$IPrimaryClassNodeOperation.doPhaseOperation(CompilationUnit.java:942) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:671) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:635) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:389) at groovy.lang.GroovyClassLoader.lambda$parseClass$3(GroovyClassLoader.java:332) at groovy.lang.GroovyClassLoader$$Lambda$1705.418737519.provide(Unknown Source:-1) at org.codehaus.groovy.runtime.memoize.StampedCommonCache.compute(StampedCommonCache.java:163) at org.codehaus.groovy.runtime.memoize.StampedCommonCache.getAndPut(StampedCommonCache.java:154) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:330)

[mcculls]

Hi @jarasez - we're currently improving our startup approach to reduce overhead and avoid this kind of issue, I'll update this issue once we have something to share.

In the meantime could you try this system property as a workaround:

-Ddd.resolver.use.loadclass=false

This will stop the tracer from loading classes directly when it's not able to find the class-file resource, which should avoid that stack trace.

An alternative workaround (assuming you're on the latest tracer release) would be to exclude the class-loaders involved in that stack trace from any matching:

-Ddd.trace.classloaders.exclude='groovy.lang.GroovyClassLoader,groovy.lang.GroovyClassLoader$InnerLoader'

[jarasez]

@mcculls Thanks for the response. The second version works only if I remove the quotes. The issue is that now I cannot see any profiles for the groovy scripts and we need them in order to profile memory/cpu. Our use case is the following: we have a java application that loads groovy scripts from the db and then executes them. The agent version: dd-java-agent-0.67.0.jar doesn't have that deadlock issue. With the latest versions: dd-java-agent-0.98.1.jar, dd-java-agent-0.99.0.jar I get the issue. Is there something else I can do, or should I use keep using agent version 0.67.0?

[mcculls]

OK, the reason 0.67.0 works is because it doesn't have the change to fall back to loadClass when it can't find the class-file resource (#2174) which went in to 0.70.0 - the first option -Ddd.resolver.use.loadclass=false turns off that feature so you could use that to workaround the issue without needing to revert to 0.67.0.

The second option excludes all classes under a given classloader - since you want to trace some classes from that classloader I would suggest just using the first option, which should avoid the issue while still allowing some classes from that classloader to be traced (ie. the ones whose class-file resources are visible.)

[jarasez]

Thanks @mcculls. It works, however in the Profiling I am not seeing much details about those scripts, as in the image below

This is the flamegraph for agent version 0.67.0.

Is there anything else I can do? Thanks.