Amazon Corretto Java 8 crashes since trace agent v1.56.0, v1.55.0 was good

[RedVortex]

Tracer Version(s)

1.56.0

Java Version(s)

1.8.0_462

JVM Vendor

Amazon Corretto

Bug Report

We started experiencing errors like this in our apps since yesterday after we started pulling version 1.56.0. This was not happening in version 1.55.0

Some of our apps run under Tomcat 8.5.100 and Amazon Corretto 1.8.0_462

This problem usually happens when the application is under load, else it is rare. But under load it may happen one time after the other...

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fb5102ce122, pid=1, tid=0x00007fb4e4760b38
#
# JRE version: OpenJDK Runtime Environment (8.0_462-b08) (build 1.8.0_462-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.462-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x2ce122]  Annotations::make_java_array(Array<unsigned char>*, Thread*)+0x72
#
# Core dump written. Default location: //core or core.1
#
# An error report file with more information is saved as:
# //hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-8/issues/

It is sometimes different but always a SIGSEV

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f5ee81c2d70, pid=1, tid=0x00007f5eca1fcb38
#
# JRE version: OpenJDK Runtime Environment (8.0_462-b08) (build 1.8.0_462-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.462-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0xa73d70]  Symbol::increment_refcount()+0x0
#
# Core dump written. Default location: //core or core.1
#
# An error report file with more information is saved as:
# //hs_err_pid1.log
#
# Compiler replay data is saved as:
# //replay_pid1.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-8/issues/

There was no code change (JSP) or any other change in the applications beside the datadog trace jar.

We re-deployed all our affected applications with 1.55.0 and the issue disappear.

Expected Behavior

Our application should not crash on version 1.56.0 of the trace java agent, this was stable in version 1.55.0, reverting to 1.55.0 fixes the issue.

Reproduction Code

No response

[mcculls]

Hi @RedVortex - could you open a support ticket and attach the full hs_err file as well as the core dump - thanks!

[mcculls]

@RedVortex also if you can recreate this in a staging environment, it would be useful to find out if any of these settings help:

DD_CODE_ORIGIN_FOR_SPANS_ENABLED=false

DD_UNSAFE_CLASS_INJECTION=true

DD_VISITOR_CLASS_PARSING=true

Each toggles a feature back to its state in v1.55.0, so if any of them do have an effect that would help focus the investigation.

[selatotal]

The same is happing with me, in the same scenario. Creating a Docker image with lastest version of Correto Java 8 Alpine and running an application with dd-java-tracer in latest-version, the problem is occurring. Fixing the 1.55.0 version is OK

[RedVortex]

Hi @RedVortex - could you open a support ticket and attach the full hs_err file as well as the core dump - thanks!

I believe someone in my team did that today after I have them the hs_err. I couldn't get the core dump however.

[RedVortex]

@RedVortex also if you can recreate this in a staging environment, it would be useful to find out if any of these settings help:

DD_CODE_ORIGIN_FOR_SPANS_ENABLED=false

DD_UNSAFE_CLASS_INJECTION=true

DD_VISITOR_CLASS_PARSING=true

Each toggles a feature back to its state in v1.55.0, so if any of them do have an effect that would help focus the investigation.

We cannot recreate easily this situation unless we do it in production and since this create outages we cannot retry this unfortunately for now.

[mcculls]

Hi @RedVortex - understood, from the data available at the moment the suggested workarounds would be to either revert to v1.55.0 or turn off the code-origin feature in v1.56.0 with:

DD_CODE_ORIGIN_FOR_SPANS_ENABLED=false

We haven't yet recreated this locally, but we have a potential solution to hand.

[mcculls]

Hi @RedVortex we just released v1.56.1 which should address the reported issue.