dd-trace-java
dd-trace-java copied to clipboard
Published
20 hours ago •
DataDog
DataDog
1.49.0 on arm64 crashes on libjavaProfiler
Tracer Version(s)
1.49.0
Java Version(s)
OpenJDK 64-Bit Server VM Temurin-24.0.1+9 (build 24.0.1+9, mixed mode, sharing)
JVM Vendor
Eclipse Adoptium / Temurin
Bug Report
With the upgrade we've encountered the error on arm64 (Graviton4) machines for our usual workloads:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000ffff9bdb4198, pid=1, tid=62
#
# JRE version: OpenJDK Runtime Environment Temurin-24.0.1+9 (24.0.1+9) (build 24.0.1+9)
# Java VM: OpenJDK 64-Bit Server VM Temurin-24.0.1+9 (24.0.1+9, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-aarch64)
# Problematic frame:
# C [ld-musl-aarch64.so.1+0x20198] __stack_chk_fail+0x4
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# JFR recording file will be written. Location: //hs_err_pid1.jfr
#
# If you would like to submit a bug report, please visit:
# https://github.com/adoptium/adoptium-support/issues
#
--------------- S U M M A R Y ------------
Command Line: -XX:MaxRAMPercentage=70.0 -XX:InitialRAMPercentage=70.0 -XX:-UseCompressedOops -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9900 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=9900 -Djava.rmi.server.hostname=10.247.151.45 -Ddd.profiling.enabled=true -XX:+PrintCodeCache -XX:ProfiledCodeHeapSize=512m -XX:NonProfiledCodeHeapSize=1024m -XX:NonNMethodCodeHeapSize=32m -XX:ReservedCodeCacheSize=1568m -XX:+UseZGC -XX:+ZGenerational -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/coredump/purevanilla-server-949c6996c-5w8j4 -XX:OnError=/app/scripts/jvm-crash-handler.sh -javaagent:/app/external/dd-java-agent.jar -Ddd.integration.throwables.enabled=false -Ddd.integration.zio.experimental.enabled=true -Ddd.trace.otel.enabled=true -Dotel.traces.sampler=always_on -Duser.timezone=Asia/Seoul -Dio.netty.allocator.numHeapArenas=0 --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.sql/java.sql=ALL-UNNAMED --enable-native-access=ALL-UNNAMED --sun-misc-unsafe-memory-access=allow -XX:+UnlockExperimentalVMOptions -XX:+UseCompactObjectHeaders com.devsisters.ck.CkMain
Host: AArch64, 16 cores, 55G, Alpine Linux v3.21
Time: Fri May 16 14:42:01 2025 KST elapsed time: 422.582783 seconds (0d 0h 7m 2s)
--------------- T H R E A D ---------------
Current thread (0x0000fffb270db210): JavaThread "C1 CompilerThread0" daemon [_thread_in_native, id=62, stack(0x0000fffb22c02000,0x0000fffb22e00a90) (2042K)]
Current CompileTask:
C1:422582 98558 3 com.devsisters.ck.program.kingdom.PaymentPrograms::mkPayment$$anonfun$1$$anonfun$1$$anonfun$3$$anonfun$1 (3039 bytes)
Stack: [0x0000fffb22c02000,0x0000fffb22e00a90], sp=0x0000fffb22df7ed0, free space=2007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [ld-musl-aarch64.so.1+0x20198] __stack_chk_fail+0x4
C [libjavaProfiler-dd-tmp2294916980829943319.so+0x50298] Profiler::recordSample(void*, unsigned long long, int, int, unsigned int, Event*)+0x114
C [libjavaProfiler-dd-tmp2294916980829943319.so+0x285d0] CTimer::signalHandler(int, siginfo_t*, void*)+0x154
C [ld-musl-aarch64.so.1+0x51b14] __setjmp+0x38
V [libjvm.so+0x4992c0] Instruction::state_values_do(ValueVisitor*)+0x20
V [libjvm.so+0x4992c0] Instruction::state_values_do(ValueVisitor*)+0x20
V [libjvm.so+0x495740] SubstitutionResolver::block_do(BlockBegin*)+0x50
V [libjvm.so+0x49a37c] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0x4c
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a428] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xf8
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
V [libjvm.so+0x49a404] BlockBegin::iterate_preorder(GrowableArray<bool>&, BlockClosure*)+0xd4
...<more frames>...
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
Registers:
R0=0x0000000000000000
R1=0x0000ffff9ad80404
R2=0x0000fffb22df7ed0
R3=0x0000fffb22df8358
R4=0x000000000000003e
R5=0x0000fffb22df8358
R6=0x0000fffb22df8374
R7=0x0000000000279f1e
R8=0xcefa665aa48200d9
R9=0x0000ffff9ad80404
R10=0x0000000000000018
R11=0x0000000000004067
R12=0x0000ffff9ad80330
R13=0x0000ffff9ad80444
R14=0x0000000000000000
R15=0x0000000000000000
R16=0x0000fffb1f208d50
R17=0x0000ffff9bdb4194
R18=0x0000fffb24866ae0
R19=0x0000fffae6bcf650
R20=0x0000fffb1fa88a50
R21=0x0000000000000080
R22=0x0000ffff9ad80404
R23=0x0000000000000000
R24=0x0000fffb22df82d8
R25=0x00000000fffffff5
R26=0x0000fffb1cba7a54
R27=0x0000fffb22dffea0
R28=0x0000fffb22df84e0
R29=0x0000fffb22df82e0
R30=0x0000fffb1f169100
Register to memory mapping:
R0 =0x0 is null
R1 =0x0000ffff9ad80404: <offset 0x000000000049a404> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
R2 =0x0000fffb22df7ed0 is pointing into the stack for thread: 0x0000fffb270db210
R3 =0x0000fffb22df8358 is pointing into the stack for thread: 0x0000fffb270db210
R4 =0x000000000000003e is an unknown value
R5 =0x0000fffb22df8358 is pointing into the stack for thread: 0x0000fffb270db210
R6 =0x0000fffb22df8374 is pointing into the stack for thread: 0x0000fffb270db210
R7 =0x0000000000279f1e is an unknown value
R8 =0xcefa665aa48200d9 is an unknown value
R9 =0x0000ffff9ad80404: <offset 0x000000000049a404> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
R10=0x0000000000000018 is an unknown value
R11=16487 is a compressed pointer to class: 0x000001f801019c00
net.bytebuddy.jar.asm.ModuleWriter {0x000001f801019c00}
- instance size: 15
- klass size: 78
- access: final synchronized
- flags: has_nonstatic_fields should_verify_class has_localvariable_table
- state: loaded
- name: 'net/bytebuddy/jar/asm/ModuleWriter'
- super: 'net/bytebuddy/jar/asm/ModuleVisitor'
- sub:
- arrays: null
- methods: Array<T>(0x0000fffb23494ff8)
- method ordering: Array<T>(0x000001f8007861f0)
- local interfaces: Array<T>(0x000001f80070d4e8)
- trans. interfaces: Array<T>(0x000001f80070d4e8)
- secondary supers: Array<T>(0x000001f80070d4d0)
- hash_slot: 16
- secondary bitmap: 0x0000000000000000
- constants: constant pool [158] {0x0000fffb23494a80} for 'net/bytebuddy/jar/asm/ModuleWriter'
- class loader data: loader data: 0x0000fffb20c56490 for instance a 'datadog/trace/bootstrap/DatadogClassLoader'{0x000004004c400828}
- source file: 'ModuleWriter.java'
- inner classes: Array<T>(0x000001f80070d4e0)
- nest members: Array<T>(0x000001f80070d4e0)
- permitted subclasses: Array<T>(0x000001f80070d4e0)
- java mirror: a 'java/lang/Class'{0x000004015dda5238} = 'net/bytebuddy/jar/asm/ModuleWriter'
- vtable length 14 (start addr: 0x000001f801019de0)
- itable length 2 (start addr: 0x000001f801019e50)
- ---- static fields (0 words):
- ---- non-static fields (14 words):
- protected final 'api' 'I' @8
- protected 'mv' 'Lnet/bytebuddy/jar/asm/ModuleVisitor;' @16
- private final 'moduleNameIndex' 'I' @12
- private final 'moduleFlags' 'I' @24
- private final 'moduleVersionIndex' 'I' @28
- private 'requiresCount' 'I' @32
- private 'exportsCount' 'I' @36
- private 'opensCount' 'I' @40
- private 'usesCount' 'I' @44
- private 'providesCount' 'I' @48
- private 'packageCount' 'I' @52
- private 'mainClassIndex' 'I' @56
- private final 'symbolTable' 'Lnet/bytebuddy/jar/asm/SymbolTable;' @64
- private final 'requires' 'Lnet/bytebuddy/jar/asm/ByteVector;' @72
- private final 'exports' 'Lnet/bytebuddy/jar/asm/ByteVector;' @80
- private final 'opens' 'Lnet/bytebuddy/jar/asm/ByteVector;' @88
- private final 'usesIndex' 'Lnet/bytebuddy/jar/asm/ByteVector;' @96
- private final 'provides' 'Lnet/bytebuddy/jar/asm/ByteVector;' @104
- private final 'packageIndex' 'Lnet/bytebuddy/jar/asm/ByteVector;' @112
- non-static oop maps: 16-16 64-112
R12=0x0000ffff9ad80330: <offset 0x000000000049a330> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
R13=0x0000ffff9ad80444: <offset 0x000000000049a444> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
R14=0x0 is null
R15=0x0 is null
R16=0x0000fffb1f208d50: <offset 0x00000000000eed50> in /tmp/ddprof_root/pid_1/scratch/libjavaProfiler-dd-tmp2294916980829943319.so at 0x0000fffb1f11a000
R17=0x0000ffff9bdb4194: __stack_chk_fail+0x0000000000000000 in /lib/ld-musl-aarch64.so.1 at 0x0000ffff9bd94000
R18={method} {0x0000fffb24866ae0} '<init>' '(Ljava/lang/Object;)V' in 'scala/MatchError'
R19=0x0000fffae6bcf650 points into unknown readable memory: 0x00000000fffffff5 | f5 ff ff ff 00 00 00 00
R20=0x0000fffb1fa88a50 points into unknown readable memory: 0x0000000000000001 | 01 00 00 00 00 00 00 00
R21=0x0000000000000080 is an unknown value
R22=0x0000ffff9ad80404: <offset 0x000000000049a404> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
R23=0x0 is null
R24=0x0000fffb22df82d8 is pointing into the stack for thread: 0x0000fffb270db210
R25=0x00000000fffffff5 is an unknown value
R26=0x0000fffb1cba7a54 points into unknown readable memory: 5f 5a 4e 31
R27=0x0000fffb22dffea0 is pointing into the stack for thread: 0x0000fffb270db210
R28=0x0000fffb22df84e0 is pointing into the stack for thread: 0x0000fffb270db210
R29=0x0000fffb22df82e0 is pointing into the stack for thread: 0x0000fffb270db210
R30=0x0000fffb1f169100: <offset 0x000000000004f100> in /tmp/ddprof_root/pid_1/scratch/libjavaProfiler-dd-tmp2294916980829943319.so at 0x0000fffb1f11a000
R31=0x0000fffb22df7ed0 is pointing into the stack for thread: 0x0000fffb270db210
Top of Stack: (sp=0x0000fffb22df7ed0)
0x0000fffb22df7ed0: 0000fffb1cba7a54 0000ffff9addf060 Tz......`.......
0x0000fffb22df7ee0: 0000ffff9ad7f2c0 0000ffff9ad7b740 ........@.......
0x0000fffb22df7ef0: 0000ffff9ad8037c 0000ffff9ad80404 |...............
0x0000fffb22df7f00: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7f10: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7f20: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7f30: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7f40: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7f50: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7f60: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7f70: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7f80: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7f90: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7fa0: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7fb0: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7fc0: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7fd0: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df7fe0: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df7ff0: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df8000: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df8010: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df8020: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df8030: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df8040: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df8050: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df8060: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df8070: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df8080: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df8090: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df80a0: 0000ffff9ad80428 0000ffff9ad80404 (...............
0x0000fffb22df80b0: 0000ffff9ad80404 0000ffff9ad80404 ................
0x0000fffb22df80c0: 0000ffff9ad80428 0000ffff9ad80404 (...............
Instructions: (pc=0x0000ffff9bdb4198)
0x0000ffff9bdb4098: f3 53 41 a9 fd 7b c3 a8 00 02 1f d6 fd 7b bc a9
0x0000ffff9bdb40a8: fd 03 00 91 f5 5b 02 a9 56 d0 3b d5 d6 22 03 d1
0x0000ffff9bdb40b8: f7 1b 00 f9 c0 62 40 f9 17 00 40 f9 b7 02 00 b4
0x0000ffff9bdb40c8: 01 05 00 d0 f3 53 01 a9 34 00 80 d2 33 44 41 f9
0x0000ffff9bdb40d8: 61 8a 40 a9 15 78 74 f8 94 06 00 91 e0 03 15 aa
0x0000ffff9bdb40e8: ca e0 ff 97 62 0e 41 a9 01 00 80 52 a0 02 02 8b
0x0000ffff9bdb40f8: 62 00 02 cb 29 e1 ff 97 73 02 40 f9 ff 02 14 eb
0x0000ffff9bdb4108: 63 00 00 54 c0 62 40 f9 f2 ff ff 17 f3 53 41 a9
0x0000ffff9bdb4118: f7 1b 40 f9 f5 5b 42 a9 fd 7b c4 a8 c0 03 5f d6
0x0000ffff9bdb4128: 00 02 00 b4 e1 03 00 aa fd 7b bf a9 02 01 80 d2
0x0000ffff9bdb4138: e0 04 00 f0 00 bc 47 f9 fd 03 00 91 b3 e0 ff 97
0x0000ffff9bdb4148: e0 04 00 f0 00 bc 47 f9 41 d0 3b d5 1f 04 00 39
0x0000ffff9bdb4158: 00 00 40 f9 20 00 1f f8 fd 7b c1 a8 c0 03 5f d6
0x0000ffff9bdb4168: e0 04 00 f0 00 bc 47 f9 a1 cd 89 d2 c1 38 a8 f2
0x0000ffff9bdb4178: 01 7c 01 9b 01 00 00 f9 1f 04 00 39 41 d0 3b d5
0x0000ffff9bdb4188: 00 00 40 f9 20 00 1f f8 c0 03 5f d6 00 00 80 d2
=>0x0000ffff9bdb4198: 1f 00 00 39 00 7d 20 d4 fd 7b be a9 e0 04 00 f0
0x0000ffff9bdb41a8: 00 d4 47 f9 fd 03 00 91 f3 0b 00 f9 13 00 40 f9
0x0000ffff9bdb41b8: 1f 00 00 f9 73 00 00 b5 06 00 00 14 93 00 00 94
0x0000ffff9bdb41c8: 60 02 40 f9 01 00 80 d2 73 22 00 91 80 ff ff b5
0x0000ffff9bdb41d8: f3 0b 40 f9 00 00 80 52 fd 7b c2 a8 c0 03 5f d6
0x0000ffff9bdb41e8: fd 7b bd a9 a1 07 80 52 fd 03 00 91 f3 53 01 a9
0x0000ffff9bdb41f8: f4 03 00 aa f5 13 00 f9 9c f9 00 94 15 00 14 eb
0x0000ffff9bdb4208: e0 00 00 54 00 00 40 39 a0 00 00 35 e0 04 00 f0
0x0000ffff9bdb4218: 00 d4 47 f9 13 00 40 f9 f3 00 00 b5 00 00 80 d2
0x0000ffff9bdb4228: f5 13 40 f9 f3 53 41 a9 fd 7b c3 a8 c0 03 5f d6
0x0000ffff9bdb4238: 73 22 00 91 61 02 40 f9 e2 03 15 aa e0 03 14 aa
0x0000ffff9bdb4248: e1 fe ff b4 cd fa 00 94 40 ff ff 35 60 02 40 f9
0x0000ffff9bdb4258: 01 68 75 38 3f f4 00 71 c1 fe ff 54 b5 06 00 91
0x0000ffff9bdb4268: 00 00 15 8b ef ff ff 17 fd 7b bb a9 fd 03 00 91
0x0000ffff9bdb4278: f7 63 03 a9 f8 03 02 aa f7 04 00 f0 f7 d6 47 f9
0x0000ffff9bdb4288: f3 53 01 a9 14 00 80 d2 f5 5b 02 a9 f6 03 00 aa
Stack slot to memory mapping:
stack at sp + 0 slots: 0x0000fffb1cba7a54 points into unknown readable memory: 5f 5a 4e 31
stack at sp + 1 slots: 0x0000ffff9addf060: <offset 0x00000000004f9060> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
stack at sp + 2 slots: 0x0000ffff9ad7f2c0: <offset 0x00000000004992c0> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
stack at sp + 3 slots: 0x0000ffff9ad7b740: <offset 0x0000000000495740> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
stack at sp + 4 slots: 0x0000ffff9ad8037c: <offset 0x000000000049a37c> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
stack at sp + 5 slots: 0x0000ffff9ad80404: <offset 0x000000000049a404> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
stack at sp + 6 slots: 0x0000ffff9ad80404: <offset 0x000000000049a404> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
stack at sp + 7 slots: 0x0000ffff9ad80404: <offset 0x000000000049a404> in /opt/java/openjdk/lib/server/libjvm.so at 0x0000ffff9a8e6000
Lock stack of current Java thread (top to bottom):
The full log may be provided privately if needed.
Expected Behavior
No compiler crashes expected.
Reproduction Code
No response
Hi @nedsociety - the stacktrace is very deep, due to some recursion in JVM and the extra few frames added by the profiler stackwalker will hit stack smashing. Musl based system are particularly sensitive to this because the stack size is in general smaller:
- https://wiki.musl-libc.org/functional-differences-from-glibc.html#thread-stack-size 
- https://github.com/yaegashi/muslstack 
- https://edu.chainguard.dev/chainguard/chainguard-images/about/images-compiled-programs/glibc-vs-musl/ 
- https://www.madetech.com/blog/alpine-linux-for-docker/
The immediate remedy would be increasing the stack size for your system.
@jbachorik Thanks for the pointers! I'll try them and report back on the next week.
Nope, increasing the native stack size from 128 Ki to 16 Mi didn't change the outcome much. This didn't happen in x86_64 version of JVM even without changing the stack size.
/# muslstack $(which java)
/opt/java/openjdk/bin/java: stackSize: 0x1000000
EDIT: maybe -XX:+UseCompactObjectHeaders might be triggering this issue? I'll try again without it.
EDIT: maybe -XX:+UseCompactObjectHeaders might be triggering this issue? I'll try again without it.
Again nope, turning off UseCompactObjectHeaders neither helped. I'm out of clues atm.
Hm, this is surprising ...
Can you disable the signal based profiler by -Ddd.profiling.ddprof.enabled=false - we will fall back to JFR only. It is not clear why the profiler should cause infinite recursion in JIT - we are definitely not modifying any code there. So, by disabling the signal based profiler we could verify whether this is because of that or something else is happening.
Thanks for your patience!
@jbachorik We've just confirmed that adding -Ddd.profiling.ddprof.enabled=false workarounds the problem! The crashes are gone.
By the way what do we lose in APM by disabling the signal based profiler?
@nedsociety Mostly the precision in the attributed CPU time (from JFR we need to estimate it on the recorder thread CPU load) and precision in tthe trace-to-profile matching.
@jbachorik Thanks! It'll take some time to confirm it though. I'll be able to get back within a few weeks.
Meanwhile you may close the issue if you wish. I'll reopen a new one if it's not resolved.
@jbachorik Quite a late report but we've confirmed that the latest version (1.55.0) does have this issue fixed. Thanks!