dd-trace-java
dd-trace-java copied to clipboard
JVM Instability with DD_PROFILING_DDPROF_ENABLED=true
Tracer Version(s)
1.46.1
Java Version(s)
17.0.14
JVM Vendor
Amazon Corretto
Bug Report
When setting DD_PROFILING_DDPROF_ENABLED=true, the JVM becomes unstable. Here are the details:
- When checking profiles for allocation,
com/datadog/profiling/controller/openjdk/events/SmapEntryFactory.javaandjava/util/regex/Matcher.javaused within it occupy a large portion (see attached image). - The buffer pool is heavily consumed, with
jvm.buffer_pool.direct.capacityreaching three times the usual amount. - Minor GC stops occurring.
- The Eden space does not decrease.
- The Datadog Profiling uploader slows down.
- Before the server stops functioning, the profile upload to Datadog ceases.
- The issue is resolved when
DD_PROFILING_DDPROF_ENABLEDis set to false.
I suspect that the following regular expression might be causing poor performance depending on the input string: https://github.com/DataDog/dd-trace-java/blob/1c3133b647c38eb44615b00ba72b4553540b8c97/dd-java-agent/agent-profiling/profiling-controller-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/events/SmapEntryFactory.java#L36
Expected Behavior
The JVM should remain stable when DD_PROFILING_DDPROF_ENABLED=true is set, without excessive buffer pool consumption or halted minor GC.
Reproduction Code
The exact steps to reproduce this issue are unclear. Even when running multiple instances concurrently, the issue appears in some instances but not others.