dd-trace-java icon indicating copy to clipboard operation
dd-trace-java copied to clipboard

JVM Instability with DD_PROFILING_DDPROF_ENABLED=true

Open RyosukeFukushima opened this issue 8 months ago • 0 comments

Tracer Version(s)

1.46.1

Java Version(s)

17.0.14

JVM Vendor

Amazon Corretto

Bug Report

When setting DD_PROFILING_DDPROF_ENABLED=true, the JVM becomes unstable. Here are the details:

  • When checking profiles for allocation, com/datadog/profiling/controller/openjdk/events/SmapEntryFactory.java and java/util/regex/Matcher.java used within it occupy a large portion (see attached image).
  • The buffer pool is heavily consumed, with jvm.buffer_pool.direct.capacity reaching three times the usual amount.
  • Minor GC stops occurring.
  • The Eden space does not decrease.
  • The Datadog Profiling uploader slows down.
  • Before the server stops functioning, the profile upload to Datadog ceases.
  • The issue is resolved when DD_PROFILING_DDPROF_ENABLED is set to false.
Image

I suspect that the following regular expression might be causing poor performance depending on the input string: https://github.com/DataDog/dd-trace-java/blob/1c3133b647c38eb44615b00ba72b4553540b8c97/dd-java-agent/agent-profiling/profiling-controller-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/events/SmapEntryFactory.java#L36

Expected Behavior

The JVM should remain stable when DD_PROFILING_DDPROF_ENABLED=true is set, without excessive buffer pool consumption or halted minor GC.

Reproduction Code

The exact steps to reproduce this issue are unclear. Even when running multiple instances concurrently, the issue appears in some instances but not others.

RyosukeFukushima avatar Mar 24 '25 05:03 RyosukeFukushima