dd-trace-java icon indicating copy to clipboard operation
dd-trace-java copied to clipboard

OOME Notifier INFO log generated when custom script is provided for -XX:OnOutOfMemoryError

Open brentm5 opened this issue 1 year ago • 2 comments

Recently we recently started seeing Datadog agent logs regarding the OOME notifier script. Below is an example log

OOME notifier script value (/service/upload_heap_dump.sh) does not follow the expected format: <path>/dd_ome_notifier.(sh|bat) %p. OOME tracking is disabled.

We actually define our own script for the -XX:OnOutOfMemoryError action which is used to upload heap dumps and other cleanup tasks. This has caused some confusion with our teams as they believe our own implementation is then turned off even though this is simply an info log. I understand logging if something is not setup correctly, however it does not appear that this takes into account where a JVM defines its own action. Additionally, I do not see an easy way to turn off this functionality to remove the log.

brentm5 avatar Oct 04 '24 21:10 brentm5

Hi @brentm5 - this line should be emitted only on startup and prefixed with a Datadog specific component. However, if this is causing inconvenience, we will move the notification to 'debug' level (it is still useful to have a notification about why the Datadog OOME notification might not be working in case of misconfiguration).

jbachorik avatar Oct 05 '24 09:10 jbachorik

For us it would ideally be a debug level log as its current log level makes it appear that there is something wrong causing OOME handling to be disabled.

One thing that would be nice is if we could better hook our own OOME handling into this so we can get the best of both worlds. Not sure if that is something others have expressed an interest in.

brentm5 avatar Oct 08 '24 19:10 brentm5

This looks to be fixed in #7750. Thanks!

brentm5 avatar Oct 28 '24 17:10 brentm5