dd-trace-java is incompatible with java 24 / 25 AOT caching

[samuelAndalon]

Tracer Version(s)

1.53.0

Java Version(s)

24, 25

JVM Vendor

Oracle JDK

Bug Report

Hello, I am not sure if this is an issue or not, but I figured starting a conversation would be nice.

In modern JVM applications, specially under distributed systems architecture, multi releases per day are very common, the performance impact of cold JVM affects the latency and throughout of JVM applications in a catastrophic way.

Project Leyden aims to alleviate this issue, specifically AOT Cache, see:

JEP 483: Ahead-of-Time Class Loading & Linking JEP 514: Ahead-of-Time Command-Line Ergonomics

The dd-trace-java is a fantastic piece of software that allows automatic instrumentation of handful of jvm libraries, unfortunately, applications using it can't try AOT caching.

This is expected, the JEP 483, specifies that training and subsequent runs should NOT use agents:

All runs must have consistent module options on the command line, and consistent module graphs. The arguments to the -m, --module, -p, --module-path, --add-modules, and --enable-native-access options, if present, must be identical. The --add-exports, --add-opens, --add-reads, --illegal-native-access, --limit-modules, --patch-module, and --upgrade-module-path options must not be used.
All runs must not use JVMTI agents that can arbitrarily rewrite classfiles using ClassFileLoadHook.
All runs must not use JVMTI agents that call the AddToBootstrapClassLoaderSearch and

I would like to open a discussion to see what can be done here, I did some basic testing and applications using AOT cache are starting 50% faster and throughout behaves as if the JVM was already warmed up, giving peak performance right away during app startup.

Expected Behavior

I would love to be able to use AOT caching in apps using the dd-trace-java

Reproduction Code

No response

[mcculls]

Hi @samuelAndalon - this is a topic of interest to us too, as part of a wider effort to improve tracer startup time. We don't have anything to share yet, but will post any updates here when we do.

[snuderl]

Alternatively is there a way to run ddtrace without agent mode?

[samuelAndalon]

@snuderl i got the dd tracer java agent working with AOT by disabling all library instrumentations, you will loose spans and metrics but at least you will have jvm metrics.

[robertocomo]

Considering AOT Cache is an iteration of CDS, this is related: #7580

[AlessioDiFazio]

Hi, I'm also facing this issue. I saw that in our case the main issue is the Tracer. It would be great to enable teams to use AOT to boost performances.

Caused by: java.lang.LinkageError: loader constraint violation: when resolving interface method 'boolean datadog.trace.api.Tracer.addTraceInterceptor(datadog.trace.api.interceptor.TraceInterceptor)' the class loader 'app' of the current class, ****/DatadogConfiguration$TraceInterceptorsConfiguration, and the class loader 'bootstrap' for the method's defining class, datadog/trace/api/Tracer, have different Class objects for the type datadog/trace/api/interceptor/TraceInterceptor used in the signature (****.DatadogConfiguration$TraceInterceptorsConfiguration is in unnamed module of loader 'app'; datadog.trace.api.Tracer is in unnamed module of loader 'bootstrap')
	at ****.DatadogConfiguration$TraceInterceptorsConfiguration.afterPropertiesSet(DatadogConfiguration.java:98)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1873)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1822)
	... 16 common frames omitted

In my case I'm create the AOT cache via

java -Xmx4g  \
  -XX:AOTCacheOutput=layers/my-service.aot \
  -Dspring.context.exit=onRefresh \
  -Dspring.profiles.active=dev \
  --add-modules=java.instrument \
  -jar layers/my-service.jar

The --add-modules=java.instrument was my attempt to "trick" the cache generation. But clearly did not work because at service startup I get the above stack trace :(

[robertocomo]

Hey @AlessioDiFazio

I’ve actually been able to partially use AOT in production for a few weeks, together with the Datadog agent, by using the --add-modules=java.instrument

• JDK 25
• dd-java-agent-1.54.0.jar

Compile:

RUN java -Djarmode=tools -jar /webapp/target/webapp.jar extract --destination /projectname/webapp
RUN java \
  -XX:+UseCompactObjectHeaders \
  -XX:AOTCacheOutput=./projectname/webapp/webapp.aot \
  -Xlog:aot=debug:file=./projectname/webapp/webapp_aot-archive.log \
  --add-modules=java.instrument \
  -Dspring.context.exit=onRefresh \
  -jar /projectname/webapp/webapp.jar

Entrypoint:

java \
  -XX:+UseCompactObjectHeaders \
  -javaagent:/agents/dd-java-agent.jar \
  -XX:SharedArchiveFile=/projectname/webapp/webapp.aot \
  -Xlog:class+load=info:file=log/webapp_class-load.log \
  -Xlog:class+path=debug:file=log/webapp_class-path.log \
  -jar /projectname/webapp/webapp.jar

Looking at webapp_class-load.log and counting the source: shared entries, we get 77.65% of classes loaded from the archive. If we remove the Datadog agent, we get 95.58%.

With CDS, the percentage was 16.82%.

[AlessioDiFazio]

@robertocomo thanks for the suggestion, unfortunately same issue on my side. The training goes well, but the runtime fails

[robertocomo]

@AlessioDiFazio I guess you are importing dd-trace-api as a dependency, aren't you?

If so:

1. Which version are you declaring ?
2. Have you tried to force the scope to provided and relying on the agent injected by the -javaagent ?

[AlessioDiFazio]

hey @robertocomo I can't have it as "provided" otherwise I won't be able to use annotations like @Trace since the symbol won't be found.

Also tried with compile scope only, but same result as above where I get a linkage runtime error

[robertocomo]

@AlessioDiFazio We are also using this approach, and it works for us. Provided dependencies are not transitive to downstream modules, so if you have a multimodule project, you need to declare the scope provided for each module. Hoping it could help!

[mcculls]

Hi @AlessioDiFazio we've added a feature in #10166 that supports using -javaagent:dd-java-agent.jar during both AOT training and production on Java 25+, which should avoid the LinkageError you observed when using dd-trace-api as part of your application.

This feature is part of the v1.57.0 release expected this week.

During training you can now use:

-XX:AOTCacheOutput=my-app.aot -javaagent:dd-java-agent-1.57.0.jar

During production use:

-XX:AOTCache=my-app.aot -javaagent:dd-java-agent-1.57.0.jar