Out of memory error using dd-java-agent.jar

[nmadhavi33]

I am facing out of memory error with below system variables

-javaagent:/path/to-dd-java-agent.jar -Ddd.profiling.enabled=true -XX:FlightRecorderOptions=stackdepth=256 -Ddd.logs.injection=true -Ddd.trace.sample.rate=1 -Ddd.service=test-api

How to tune these parameters so that it won't throw out of memory error?

[mcculls]

Hi @nmadhavi33 - what sort of application is this and how are you deploying it? Also which version of the tracer are you using?

Are you able to share some/all of the stack trace when the out of memory happens?

If you have a support contract you could capture a heap dump on the out of memory by running the application with -XX:+HeapDumpOnOutOfMemoryError and send it to us via a support ticket.

[nmadhavi33]

I am using 0.100.0 tracer version, and this is AKKA framework java application with java 8

Exception logs

[dd.trace 2022-05-22 01:38:45:368 -0400] [OkHttp http://localhost:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile, received empty reply from http://localhost:8126/profiling/v1/input after uploading profile (Will not log errors for 5 minutes) [dd.trace 2022-05-22 01:41:18:284 -0400] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - Error while sending 131 (size=56KB) traces to the DD agent. Total: 1447725, Received: 1447725, Sent: 1447594, Failed: 131. java.net.SocketTimeoutException: timeout (Will not log errors for 5 minutes) [dd.trace 2022-05-22 01:45:58:691 -0400] [dd-trace-processor] INFO datadog.trace.agent.common.writer.ddagent.DDAgentApi - Success while sending 580 (size=248KB) traces to the DD agent. Total: 1449155, Received: 1449155, Sent: 1448174, Failed: 981. java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid6.hprof ... *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 807

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "StatsD-Sender-1" Dump file is incomplete: No space left on device Terminating due to java.lang.OutOfMemoryError: Java heap space

[mcculls]

Thanks - does the out of memory error happen when profiling is turned off? ie. -Ddd.profiling.enabled=false

I see there's a separate error above saying that it wasn't able to send profiling data to the agent. It could be that this extra profiling data (which wasn't able to be sent) is using up part of the heap and tipping your application's heap usage over the current limit. I'd suggest looking into why it wasn't able to send the profiling data to the agent.

Meanwhile you could also try increasing your -Xmx setting - once you've increased it so the out of memory error no longer occurs then you can look at the JVM heap metrics to see how much it uses over time (ie. after GCs) which would then let you set a more accurate heap limit.

The No space left on device warning is also something to look at - you could check your temporary directory to see if there's any old files that can be cleared up. The Java Flight Recorder uses temporary files to buffer some of its profiling data, which the profiler will try to clean up on exit - but I would recommend monitoring how much disk space is used over time on that host, in case you need to give it more disk space to account for the extra profiling data.