adoptium-support
adoptium-support copied to clipboard
adoptium
SIGSEGV during C2 CompileTask
Please provide a brief summary of the bug
Hi, it seems that since we started using eclipse-temurin:21-jdk-noble (21.0.5+11) in our CI (via maven:3-eclipse-temurin-21), we get random JVM crashes during execution of the C2 compiler. It's always with the same trace and always during the compilation of com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields.
I didn't find anything resembling this error in the OpenJDK JIRA nor anywhere on the internet :/
Did you test with the latest update version?
- [X] Yes
Please provide steps to reproduce where possible
Not sure how, it's quite random
Expected Results
No crash :P
Actual Results
Random crashes :)
What Java Version are you using?
OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
What is your operating system and platform?
Ubuntu 24.04.1 LTS on x86
How did you install Java?
Docker image maven:3-eclipse-temurin-21
Did it work before?
Yes, I think in the previous version (21.0.4+7) but it's hard to exactly know.
Did you test with other Java versions?
Because it's using a docker image, I don't have a way to use an older version of Java unfortunately, but I will try to find a way to do that next.
Relevant log output
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007d5f17168bd9, pid=85, tid=103
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.5+11 (21.0.5+11) (build 21.0.5+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xf0dbd9] BoolNode::Ideal(PhaseGVN*, bool)+0x19
#
# Core dump will be written. Default location: /core.%e.85.%t
#
# If you would like to submit a bug report, please visit:
# https://github.com/adoptium/adoptium-support/issues
#
--------------- S U M M A R Y ------------
Command Line: -javaagent:/cache/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/builds/server/target/jacoco-it.exec -Dmaven.repo.local=/cache/.m2/repository -Xmx8192m -XX:ActiveProcessorCount=2 -Xverify:none --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED /builds/server/target/surefire/surefirebooter9049070912728376033.jar /builds/server/target/surefire 2024-11-07T13-53-41_126-jvmRun1 surefire3272181257269677729tmp surefire_018442978085457632387tmp
Host: AMD EPYC 7B13, 16 cores, 9G, Ubuntu 24.04.1 LTS
Time: Thu Nov 7 14:07:52 2024 UTC elapsed time: 851.233062 seconds (0d 0h 14m 11s)
--------------- T H R E A D ---------------
Current thread (0x00007d5f100cb350): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=103, stack(0x00007d5eed0b2000,0x00007d5eed1b2000) (1024K)]
Current CompileTask:
C2:851233 108248 4 com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields (1097 bytes)
Stack: [0x00007d5eed0b2000,0x00007d5eed1b2000], sp=0x00007d5eed1ad6d0, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf0dbd9] BoolNode::Ideal(PhaseGVN*, bool)+0x19
V [libjvm.so+0xd5acfe] PhaseIterGVN::transform_old(Node*)+0x9e
V [libjvm.so+0x6b30e0] Conv2BNode::Ideal(PhaseGVN*, bool)+0x180
V [libjvm.so+0xd5acfe] PhaseIterGVN::transform_old(Node*)+0x9e
V [libjvm.so+0xd56c19] PhaseIterGVN::optimize()+0xf9
V [libjvm.so+0x67072f] Compile::Optimize()+0xf9f
V [libjvm.so+0x671e66] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0xf26
V [libjvm.so+0x597fbb] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x18b
V [libjvm.so+0x67833e] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9e
V [libjvm.so+0x67b538] CompileBroker::compiler_thread_loop()+0x6a8
V [libjvm.so+0x92a798] JavaThread::thread_main_inner()+0x1d8
V [libjvm.so+0xf7eeb8] Thread::call_run()+0xa8
V [libjvm.so+0xd08a8a] thread_native_entry(Thread*)+0xda
C [libc.so.6+0x9ca94]
@victornoel which version of Jackson are you usin and do you experience these crashes with Jacoco diasble?
Hi @karianna, sorry for the delay.
So what I can already tell is that with 21.0.4+7 we don't get the problem (the CI has been using it for 4-5 days).
We use Jackson 2.17.2 (and seen it happen with 2.17.1 too.
We do use jacoco (0.8.11) indeed, and I haven't had the opportunity to test without it, but I will try to set that up as soon as I can.
Thanks @victornoel - It could well be a genuine 21.0.5 bug, but it could also be a genuine 21.0.5 C2 change that jacoco needs to adjust for (this is a not uncommon problem with trivial versions of Java and tools like it). If we can eliminate jacoco then it's easire for me to get this properly looked at upstream :-)
@karianna yep, it makes sense, I will try it and come back to you. I'm not blocked by this btw
I have setup a jacoco-less CI pipeline that runs regularly using maven:3-eclipse-temurin-21, we will see how it behaves after a few days :)
Hello! I have the same issue in my company's pipeline after making a refactor on the code. Our GitHub action workers with temurin 21.0.5+11 have been upgraded on Friday and we started seeing issues right away on one of the two PRs.
It happens in 90% of the executions of the pipeline for the refactored code. Unfortunately, I can't provide it due to privacy policies, but if I am able to reproduce it, I'll let you know. Cheers!
Hereby, I attach the hs_err_pidXXXX.log stack.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f66a3d0dbd9, pid=2560, tid=2578
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.5+11 (21.0.5+11) (build 21.0.5+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xf0dbd9] BoolNode::Ideal(PhaseGVN*, bool)+0x19
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/pricing/pricing/core.2560)
#
# If you would like to submit a bug report, please visit:
# https://github.com/adoptium/adoptium-support/issues
#
--------------- S U M M A R Y ------------
Command Line: -javaagent:/home/runner/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/home/runner/work/pricing/pricing/target/jacoco.exec,excludes=**/config/*:**/liquibase/executors/*:**/service/residualvalue/*:**/PricingFeature.*:**/MigrationController.* -XX:+EnableDynamicAgentLoading /home/runner/work/pricing/pricing/target/surefire/surefirebooter-20241118154552761_3.jar /home/runner/work/pricing/pricing/target/surefire 2024-11-18T15-45-52_707-jvmRun1 surefire-20241118154552761_1tmp surefire_0-20241118154552761_2tmp
Host: AMD EPYC 7763 64-Core Processor, 8 cores, 31G, Ubuntu 22.04.5 LTS
Time: Mon Nov 18 15:47:28 2024 UTC elapsed time: 96.009229 seconds (0d 0h 1m 36s)
--------------- T H R E A D ---------------
Current thread (0x00007f669c135b30): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=2578, stack(0x00007f666d3fb000,0x00007f666d4fb000) (1024K)]
Current CompileTask:
C2:96009 56567 % 4 com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields @ 117 (1097 bytes)
Stack: [0x00007f666d3fb000,0x00007f666d4fb000], sp=0x00007f666d4f6660, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf0dbd9] BoolNode::Ideal(PhaseGVN*, bool)+0x19
V [libjvm.so+0xd5acfe] PhaseIterGVN::transform_old(Node*)+0x9e
V [libjvm.so+0x6b30e0] Conv2BNode::Ideal(PhaseGVN*, bool)+0x180
V [libjvm.so+0xd5acfe] PhaseIterGVN::transform_old(Node*)+0x9e
V [libjvm.so+0xd56c19] PhaseIterGVN::optimize()+0xf9
V [libjvm.so+0x67072f] Compile::Optimize()+0xf9f
V [libjvm.so+0x671e66] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0xf26
V [libjvm.so+0x597fbb] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x18b
V [libjvm.so+0x67833e] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9e
V [libjvm.so+0x67b538] CompileBroker::compiler_thread_loop()+0x6a8
V [libjvm.so+0x92a798] JavaThread::thread_main_inner()+0x1d8
V [libjvm.so+0xf7eeb8] Thread::call_run()+0xa8
V [libjvm.so+0xd08a8a] thread_native_entry(Thread*)+0xda
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000000000002c
Registers:
RAX=0x00007f65742d1d90, RBX=0x00007f66a431eb50, RCX=0x00007f66008123d8, RDX=0x0000000000000001
RSP=0x00007f666d4f6660, RBP=0x00007f666d4f66e0, RSI=0x00007f666d4f6a30, RDI=0x00007f65742d1d58
R8 =0x0000000000000129, R9 =0x0000000000000000, R10=0x0000000000000000, R11=0x00007f6578b843c0
R12=0x00007f666d4f6a30, R13=0x0000000000000000, R14=0x00007f666d4f6a30, R15=0x00007f66a32b6db0
RIP=0x00007f66a3d0dbd9, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Register to memory mapping:
RAX=0x00007f65742d1d90 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
RBX=0x00007f66a431eb50: <offset 0x000000000151eb50> in /usr/lib/jvm/temurin-21-jdk-amd64/lib/server/libjvm.so at 0x00007f66a2e00000
RCX=0x00007f66008123d8 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
RDX=0x0000000000000001 is an unknown value
RSP=0x00007f666d4f6660 is pointing into the stack for thread: 0x00007f669c135b30
RBP=0x00007f666d4f66e0 is pointing into the stack for thread: 0x00007f669c135b30
RSI=0x00007f666d4f6a30 is pointing into the stack for thread: 0x00007f669c135b30
RDI=0x00007f65742d1d58 points into unknown readable memory: 0x00007f66a42e4920 | 20 49 2e a4 66 7f 00 00
R8 =0x0000000000000129 is an unknown value
R9 =0x0 is null
R10=0x0 is null
R11=0x00007f6578b843c0 points into unknown readable memory: 0x00007f66a42c9900 | 00 99 2c a4 66 7f 00 00
R12=0x00007f666d4f6a30 is pointing into the stack for thread: 0x00007f669c135b30
R13=0x0 is null
R14=0x00007f666d4f6a30 is pointing into the stack for thread: 0x00007f669c135b30
R15=0x00007f66a32b6db0: <offset 0x00000000004b6db0> in /usr/lib/jvm/temurin-21-jdk-amd64/lib/server/libjvm.so at 0x00007f66a2e00000
Top of Stack: (sp=0x00007f666d4f6660)
0x00007f666d4f6660: 00007f65681a2e90 0000000000058298
0x00007f666d4f6670: 00007f66000e28a0 00007f66000e85c0
0x00007f666d4f6680: 00007f66000ea888 00007f669c133fa0
0x00007f666d4f6690: 00007f66000e85c0 00007f66000e28a0
0x00007f666d4f66a0: 00007f66000e85b0 00007f66a3ad1995
0x00007f666d4f66b0: 00007f65742d1d58 00007f66a431eb50
0x00007f666d4f66c0: 00007f666d4f6a30 00007f65742d1d58
0x00007f666d4f66d0: 00007f666d4f6a30 00007f66a32b6db0
0x00007f666d4f66e0: 00007f666d4f6720 00007f66a3b5acfe
0x00007f666d4f66f0: 00007f666d4f8a60 00007f66a3b5c1c0
0x00007f666d4f6700: 00007f666d4f6a30 00007f65742d1d58
0x00007f666d4f6710: 0000000000000000 00007f657402b810
0x00007f666d4f6720: 00007f666d4f6760 00007f66a34b30e0
0x00007f666d4f6730: 00007f6600334b18 00007f66a431eb50
0x00007f666d4f6740: 00007f666d4f6a30 00007f65f4807e90
0x00007f666d4f6750: 00007f666d4f6a30 00007f66a32b6db0
0x00007f666d4f6760: 00007f666d4f67a0 00007f66a3b5acfe
0x00007f666d4f6770: 00007f66a431e1c0 00007f666d4f8a60
0x00007f666d4f6780: 00007f666d4f6a30 00007f65f4807e90
0x00007f666d4f6790: 0000000000000135 00007f66a42ef708
0x00007f666d4f67a0: 00007f666d4f67f0 00007f66a3b56c19
0x00007f666d4f67b0: 00007f6600000000 00007f666d4f6a30
0x00007f666d4f67c0: 00007f666d4f67f0 00007f6600eabaf0
0x00007f666d4f67d0: 00007f666d4f8a60 00007f666d4f6a30
0x00007f666d4f67e0: 00007f666d4f73a0 00007f666d4f6880
0x00007f666d4f67f0: 00007f666d4f7d90 00007f66a347072f
0x00007f666d4f6800: 00007f666d4f8a60 00007f666d4f73a0
0x00007f666d4f6810: 00007f66000e28a0 00007f66000e85c0
0x00007f666d4f6820: 00007f666d4f6880 00007f66000e85c0
0x00007f666d4f6830: 00007f666d4f6a30 00007f666d4f73a0
0x00007f666d4f6840: 00007f666d4f6940 00007f669c001af0
0x00007f666d4f6850: 00007f669c133fa0 00007f6600636530
Instructions: (pc=0x00007f66a3d0dbd9)
0x00007f66a3d0dad9: 00 4c 89 e7 e8 ae 0f dc ff 48 8d 05 5f 7c 5b 00
0x00007f66a3d0dae9: 4d 89 6c 24 38 41 c7 44 24 2c 04 00 00 00 48 83
0x00007f66a3d0daf9: c0 10 49 89 04 24 66 48 8d 3d b9 a8 5d 00 66 66
0x00007f66a3d0db09: 48 e8 61 4f 39 ff 48 8b 00 48 8b 80 00 07 00 00
0x00007f66a3d0db19: 48 8b 80 80 00 00 00 48 8b 98 e0 02 00 00 49 8b
0x00007f66a3d0db29: 44 24 08 48 89 18 48 85 db 74 1b 48 8b 43 10 48
0x00007f66a3d0db39: 85 c0 74 12 8b 73 20 3b 73 24 74 68 8d 56 01 89
0x00007f66a3d0db49: 53 20 4c 89 24 f0 48 8d 05 02 80 5b 00 41 83 4c
0x00007f66a3d0db59: 24 30 10 41 c7 44 24 2c 04 18 00 00 48 83 c0 10
0x00007f66a3d0db69: 49 89 04 24 eb 04 90 45 31 e4 48 83 c4 08 4c 89
0x00007f66a3d0db79: e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 8d b8 80
0x00007f66a3d0db89: 02 00 00 31 d2 be 40 00 00 00 e8 c8 58 53 ff 49
0x00007f66a3d0db99: 89 c4 e9 11 ff ff ff 4c 89 ef ff d0 49 89 c5 e9
0x00007f66a3d0dba9: 28 ff ff ff 48 89 df e8 8b 12 dc ff 48 8b 43 10
0x00007f66a3d0dbb9: 8b 73 20 eb 87 66 90 55 48 89 e5 41 57 41 56 41
0x00007f66a3d0dbc9: 55 41 54 53 48 83 ec 58 48 8b 47 08 4c 8b 68 08
0x00007f66a3d0dbd9: 41 8b 45 2c 83 e0 7f 83 f8 40 0f 85 77 02 00 00
0x00007f66a3d0dbe9: 49 8b 45 00 49 89 fc 49 89 f6 4c 89 ef ff 10 41
0x00007f66a3d0dbf9: 89 c7 8d 80 56 ff ff ff 83 f8 01 0f 86 56 02 00
0x00007f66a3d0dc09: 00 41 8b 45 2c 25 ff 07 00 00 3d c0 04 00 00 0f
0x00007f66a3d0dc19: 94 c3 41 81 ff dd 01 00 00 0f 94 c0 08 c3 0f 85
0x00007f66a3d0dc29: 33 02 00 00 49 8b 45 08 4c 8b 48 10 4c 8b 50 08
0x00007f66a3d0dc39: 4c 89 4d c0 4d 85 d2 0f 84 1a 02 00 00 41 8b 44
0x00007f66a3d0dc49: 24 34 83 e0 fb 83 f8 02 0f 84 09 02 00 00 49 8b
0x00007f66a3d0dc59: 02 4c 89 d7 4c 89 55 b8 ff 10 4c 8b 4d c0 89 45
0x00007f66a3d0dc69: c8 49 8b 01 4c 89 cf ff 10 4c 8b 55 b8 4c 8b 4d
0x00007f66a3d0dc79: c0 41 89 c0 41 f6 42 30 10 74 24 41 f6 41 30 10
0x00007f66a3d0dc89: 75 1d 44 89 f8 83 e0 fd 83 f8 4d 74 12 41 81 f8
0x00007f66a3d0dc99: 02 01 00 00 0f 85 65 07 00 00 0f 1f 44 00 00 41
0x00007f66a3d0dca9: 83 ff 51 0f 84 46 05 00 00 41 83 ff 52 0f 85 74
0x00007f66a3d0dcb9: 01 00 00 41 f7 44 24 34 fb ff ff ff 0f 85 ad 01
0x00007f66a3d0dcc9: 00 00 83 7d c8 1d 0f 85 a3 01 00 00 41 83 f8 7c
Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007f65681a2e90 points into unknown readable memory: 0x00007f66a422cbc0 | c0 cb 22 a4 66 7f 00 00
stack at sp + 1 slots: 0x0000000000058298 is an unknown value
stack at sp + 2 slots: 0x00007f66000e28a0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
stack at sp + 3 slots: 0x00007f66000e85c0 points into unknown readable memory: 0x747361662e6d6f63 | 63 6f 6d 2e 66 61 73 74
stack at sp + 4 slots: 0x00007f66000ea888 points into unknown readable memory: 0x0000000000000035 | 35 00 00 00 00 00 00 00
stack at sp + 5 slots: 0x00007f669c133fa0 points into unknown readable memory: 0x00000007f669c107 | 07 c1 69 f6 07 00 00 00
stack at sp + 6 slots: 0x00007f66000e85c0 points into unknown readable memory: 0x747361662e6d6f63 | 63 6f 6d 2e 66 61 73 74
stack at sp + 7 slots: 0x00007f66000e28a0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
Update: when I ran the same pipeline with temurin 21.0.4+7.0.LTS the builds started passing.
@vzedano hi, since you are so "lucky" that it fails often, would you mind checking if it fails with jacoco disabled (if you use it)? Is it also using Jackson and falling on the same place as in my report? Thanks
Hi @karianna, after a week of running latest jdk 21 with jacoco disabled, I did not see the issue appear. What would be the next step then? Thanks!
@victornoel - I'd report this to the Jacoco team in that case. https://groups.google.com/g/jacoco
@karianna will do, thx for all the help! I will report back here before closing this ticket.
I created https://github.com/jacoco/jacoco/issues/1801 and also posted a message in the ML.
Hey @karianna, so I'm here acting just as a messenger (:P), it seems that jacoco people are happy to help us finding the problem, but they basically say (see https://github.com/jacoco/jacoco/issues/1801#issuecomment-2502362021) that they don't see how they could fix this in jacoco to avoid such a bug because jacoco is about bytecode enhancement.
They did say that many time in the past helped track down JDK bugs and so they can help. Luckily it seems that @Godin may have also encountered that issue at its job but it's hard to reproduce. Ideally if we can get output from a fastdebug build of the JDK, that would greatly help.
Do you know if we have such an image at adoptium? Cheers
Hi @karianna and happy new year :) any idea how we can make this move forward?
We don't have fastdebug builds readily available sadly. I'll try to build one locally and if that works I'll send you the instructions on how to do it.
OK so long story short is you need to:
- Clone the temurin-build repo
- Clone the jdk21u repo
- sudo apt-get install autoconf libcups2-dev libx11-dev libxext-dev libxrender-dev libxrandr-dev libxtst-dev libxt-dev libasound2-dev
- ./makejdk-any-platform.sh -l
--create-debug-image jdk21u
I have some more examples at the link below that identifies the same problematic frame from our Develocity instance. It occurred on both macOS and Linux.
The project uses JaCoCo 0.8.12 but does not use Jackson.
https://ge.detekt.dev/scans/failures?failures.failureClassification=verification&failures.failureMessage=Execution%20failed%20for%20task%20*%0A%3E%20Failure%20during%20test%20execution:%20Forked%20test%20JVM%20terminated%20unexpectedly%20with%20exit%20value%20134%0A%20%20Test%20Distribution%20has%20stopped%20executing%20any%20remaining%20tests%20and%20silently%20skips%20them!%0A%20%20For%20potential%20reasons%20why%20the%20test%20JVM%20exited%2C%20check%20the%20troubleshooting%20section%20at%20https:%2F%2Fgradle.com%2Fhelp%2Ftest-distribution-troubleshooting.%0A%20%20Standard%20output%20from%20JVM:%0A%20%20%20%20%20%20%23%0A%20%20%20%20%20%20%23%20A%20fatal%20error%20has%20been%20detected%20by%20the%20Java%20Runtime%20Environment:%0A%20%20%20%20%20%20%23%0A%20%20%20%20%20%20%23%20%20SIGSEGV%20(0xb)%20at%20*%0A%20%20%20%20%20%20%23%0A%20%20%20%20%20%20%23%20JRE%20version:%20OpenJDK%20Runtime%20Environment%20*%20(build%20*%0A%20%20%20%20%20%20%23%20Java%20VM:%20OpenJDK%2064-Bit%20Server%20VM%20*%20mixed%20mode%2C%20sharing%2C%20tiered%2C%20compressed%20oops%2C%20compressed%20class%20ptrs%2C%20g1%20gc%2C%20*%0A%20%20%20%20%20%20%23%20Problematic%20frame:%0A%20%20%20%20%20%20%23%20V%20%20*%20%20BoolNode::Ideal(PhaseGVN%5C*%2C%20*%0A%20%20%20%20%20%20%23%0A%20%20%20%20%20%20%23%20*%20dump%20will%20be%20written.%20*&search.relativeStartTime=P90D&search.timeZoneId=Australia%2FSydney
Turns out we do have fast debug builds - try via our API:
e.g. https://api.adoptium.net/v3/binary/latest/21/ga/linux/x64/debugimage/hotspot/normal/eclipse
Also running into the same issue. We had stable CI-builds with 21.0.4+7, crashes started with 21.0.5+11. Now with 21.0.6+7 identical crashes are happening.
Integrated the debugimage files as well:
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000797aa4d07a79, pid=4143483, tid=4145135
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.6+7 (21.0.6+7) (build 21.0.6+7-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.6+7 (21.0.6+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xf07a79] BoolNode::Ideal(PhaseGVN*, bool)+0x19
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/jenkins/jenkins-agent/workspace/sp/x/core.4143483)
#
# If you would like to submit a bug report, please visit:
# https://github.com/adoptium/adoptium-support/issues
#
--------------- S U M M A R Y ------------
Command Line: -Dorg.gradle.internal.worker.tmpdir=/home/jenkins/jenkins-agent/workspace/sp/x/build/tmp/test/work -javaagent:/home/jenkins/jenkins-agent/workspace/sp/x/build/tmp/.cache/expanded/zip_ff7bf8f04f99dc306508d1c81e47a68b/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false -Xms512m -Xmx1024m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -Djava.io.tmpdir=/home/jenkins-cloudbuild/tmp worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 3'
Host: 13th Gen Intel(R) Core(TM) i9-13900, 32 cores, 62G, Arch Linux
Time: Tue Apr 15 09:48:48 2025 CEST elapsed time: 53.944849 seconds (0d 0h 0m 53s)
--------------- T H R E A D ---------------
Current thread (0x0000797a004eb880): JavaThread "C2 CompilerThread6" daemon [_thread_in_native, id=4145135, stack(0x000079790a5e7000,0x000079790a6e7000) (1024K)]
Current CompileTask:
C2:53944 73366 4 com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields (1097 bytes)
Stack: [0x000079790a5e7000,0x000079790a6e7000], sp=0x000079790a6e2690, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf07a79] BoolNode::Ideal(PhaseGVN*, bool)+0x19 (node.hpp:960)
V [libjvm.so+0xd563ae] PhaseIterGVN::transform_old(Node*)+0x9e (phaseX.cpp:667)
V [libjvm.so+0x6b252a] Conv2BNode::Ideal(PhaseGVN*, bool)+0x17a (convertnode.cpp:81)
V [libjvm.so+0xd563ae] PhaseIterGVN::transform_old(Node*)+0x9e (phaseX.cpp:667)
V [libjvm.so+0xd52339] PhaseIterGVN::optimize()+0xf9 (phaseX.cpp:1045)
V [libjvm.so+0x66feb9] Compile::Optimize()+0xfc9 (compile.cpp:1856)
V [libjvm.so+0x671616] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0xf36 (compile.cpp:851)
V [libjvm.so+0x59859b] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x18b (c2compiler.cpp:119)
V [libjvm.so+0x677a9e] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9e (compileBroker.cpp:2272)
V [libjvm.so+0x67ac88] CompileBroker::compiler_thread_loop()+0x6a8 (compileBroker.cpp:1951)
V [libjvm.so+0x928c08] JavaThread::thread_main_inner()+0x1d8 (javaThread.cpp:717)
V [libjvm.so+0xf78778] Thread::call_run()+0xa8 (thread.cpp:221)
V [libjvm.so+0xd042ba] thread_native_entry(Thread*)+0xda (os_linux.cpp:790)
C [libc.so.6+0x9570a]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000000000002c
Registers:
RAX=0x0000797898019aa0, RBX=0x0000797aa53189d0, RCX=0x00007978984ddf88, RDX=0x0000000000000001
RSP=0x000079790a6e2690, RBP=0x000079790a6e2710, RSI=0x000079790a6e2aa0, RDI=0x0000797898019a68
R8 =0x0000000000000129, R9 =0x0000000000000000, R10=0x0000000000000000, R11=0x00007978a4267778
R12=0x000079790a6e2aa0, R13=0x0000000000000000, R14=0x000079790a6e2aa0, R15=0x0000797aa42b7a10
RIP=0x0000797aa4d07a79, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
XMM[0]=0x0000000000000000 0x0000000200000002
XMM[1]=0x00007978a14dcd30 0x00007978a14dcb30
XMM[2]=0x00007978a14dcb30 0x00007978a14d63f0
XMM[3]=0x00007978a14dcb30 0x00007978a14d63f0
XMM[4]=0x0000000000000000 0x0000000000000000
XMM[5]=0x0000000000000000 0x0000000000000000
XMM[6]=0x00007978a14dcb30 0x00007978a14d63f0
XMM[7]=0x0000000000000000 0x0000000000000000
XMM[8]=0x0000000000000000 0x0000000000000000
XMM[9]=0x0000000000000000 0x4008000000000000
XMM[10]=0x0000000000000000 0x46293e5939a08cea
XMM[11]=0x0000000000000000 0x4376345785d8a000
XMM[12]=0x0000797a0930e540 0x0000797a0930e4e0
XMM[13]=0x0000000000000000 0x0000797a0930e9c0
XMM[14]=0x0000000000000000 0x0000797a0930ea80
XMM[15]=0x0000000000000000 0x0000000000000000
MXCSR=0x00001fa0
Register to memory mapping:
RAX=0x0000797898019aa0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
RBX=0x0000797aa53189d0: <offset 0x00000000015189d0> in /home/jenkins/jenkins-agent/tools/hudson.model.JDK/OpenJDK-21.0.6_7-debugimage/jdk-21.0.6+7/lib/server/libjvm.so at 0x0000797aa3e00000
RCX=0x00007978984ddf88 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
RDX=0x0000000000000001 is an unknown value
RSP=0x000079790a6e2690 is pointing into the stack for thread: 0x0000797a004eb880
RBP=0x000079790a6e2710 is pointing into the stack for thread: 0x0000797a004eb880
RSI=0x000079790a6e2aa0 is pointing into the stack for thread: 0x0000797a004eb880
RDI=0x0000797898019a68 points into unknown readable memory: 0x0000797aa52de920 | 20 e9 2d a5 7a 79 00 00
R8 =0x0000000000000129 is an unknown value
R9 =0x0 is null
R10=0x0 is null
R11=0x00007978a4267778 points into unknown readable memory: 0x0000797aa52c3900 | 00 39 2c a5 7a 79 00 00
R12=0x000079790a6e2aa0 is pointing into the stack for thread: 0x0000797a004eb880
R13=0x0 is null
R14=0x000079790a6e2aa0 is pointing into the stack for thread: 0x0000797a004eb880
R15=0x0000797aa42b7a10: <offset 0x00000000004b7a10> in /home/jenkins/jenkins-agent/tools/hudson.model.JDK/OpenJDK-21.0.6_7-debugimage/jdk-21.0.6+7/lib/server/libjvm.so at 0x0000797aa3e00000
Top of Stack: (sp=0x000079790a6e2690)
0x000079790a6e2690: 000079789c1b3f98 0000000000068248
0x000079790a6e26a0: 00007978a14d63f0 00007978a14dcb30
0x000079790a6e26b0: 00007978a14de3d8 0000797a004ebfc0
0x000079790a6e26c0: 00007978a14dcb30 00007978a14d63f0
0x000079790a6e26d0: 00007978a14dcb20 0000797aa4acd2c5
0x000079790a6e26e0: 0000797898019a68 0000797aa53189d0
0x000079790a6e26f0: 000079790a6e2aa0 0000797898019a68
0x000079790a6e2700: 000079790a6e2aa0 0000797aa42b7a10
0x000079790a6e2710: 000079790a6e2750 0000797aa4b563ae
0x000079790a6e2720: 000079790a6e4ad0 0000797aa4b57880
0x000079790a6e2730: 000079790a6e2aa0 0000797898019a68
0x000079790a6e2740: 0000000000000000 00007978aca4feb0
0x000079790a6e2750: 000079790a6e2790 0000797aa44b252a
0x000079790a6e2760: 00007979fc0318d0 0000797aa53189d0
0x000079790a6e2770: 000079790a6e2aa0 00007979b53c1b40
0x000079790a6e2780: 000079790a6e2aa0 0000797aa42b7a10
0x000079790a6e2790: 000079790a6e27d0 0000797aa4b563ae
0x000079790a6e27a0: 0000000000000000 000079790a6e4ad0
0x000079790a6e27b0: 000079790a6e2aa0 00007979b53c1b40
0x000079790a6e27c0: 0000000000000159 0000797aa52e9560
0x000079790a6e27d0: 000079790a6e2820 0000797aa4b52339
0x000079790a6e27e0: 0000797900000000 000079790a6e2aa0
0x000079790a6e27f0: 000079790a6e2820 00007978acac3540
0x000079790a6e2800: 000079790a6e4ad0 0000797a004ebfc0
0x000079790a6e2810: 000079790a6e2aa0 000079790a6e3410
0x000079790a6e2820: 000079790a6e3e00 0000797aa446feb9
0x000079790a6e2830: 0000002800000000 000079790a6e4ad0
0x000079790a6e2840: 000079790a6e3410 0000797a004ebfc0
0x000079790a6e2850: 00007978a14d63f0 00007978a14dcb30
0x000079790a6e2860: 00007978a14dcb30 000079790a6e2aa0
0x000079790a6e2870: 0000797a004ebfc0 000079790a6e29b0
0x000079790a6e2880: 0000797a9c005e60 00000000000382c0
Instructions: (pc=0x0000797aa4d07a79)
0x0000797aa4d07979: 89 6c 24 38 41 c7 44 24 2c 04 00 00 00 48 83 c0
0x0000797aa4d07989: 10 49 89 04 24 66 48 8d 3d 2a aa 5d 00 66 66 48
0x0000797aa4d07999: e8 72 ad 39 ff 48 8b 00 48 8b 80 00 07 00 00 48
0x0000797aa4d079a9: 8b 80 80 00 00 00 48 8b 98 e0 02 00 00 49 8b 44
0x0000797aa4d079b9: 24 08 48 89 18 48 85 db 74 1b 48 8b 43 10 48 85
0x0000797aa4d079c9: c0 74 12 8b 73 20 3b 73 24 74 69 8d 56 01 89 53
0x0000797aa4d079d9: 20 4c 89 24 f0 48 8d 05 73 81 5b 00 41 83 4c 24
0x0000797aa4d079e9: 30 10 41 c7 44 24 2c 04 18 00 00 48 83 c0 10 49
0x0000797aa4d079f9: 89 04 24 eb 05 66 90 45 31 e4 48 83 c4 08 4c 89
0x0000797aa4d07a09: e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 8d b8 80
0x0000797aa4d07a19: 02 00 00 31 d2 be 40 00 00 00 e8 48 c7 53 ff 49
0x0000797aa4d07a29: 89 c4 e9 10 ff ff ff 4c 89 ef ff d0 49 89 c5 e9
0x0000797aa4d07a39: 27 ff ff ff 48 89 df e8 4b 2d dc ff 48 8b 43 10
0x0000797aa4d07a49: 8b 73 20 e9 83 ff ff ff 90 66 66 2e 0f 1f 84 00
0x0000797aa4d07a59: 00 00 00 00 0f 1f 00 55 48 89 e5 41 57 41 56 41
0x0000797aa4d07a69: 55 41 54 53 48 83 ec 58 48 8b 47 08 4c 8b 68 08
0x0000797aa4d07a79: 41 8b 45 2c 83 e0 7f 83 f8 40 0f 85 37 01 00 00
0x0000797aa4d07a89: 49 8b 45 00 49 89 fc 49 89 f6 4c 89 ef ff 10 89
0x0000797aa4d07a99: c3 8d 80 56 ff ff ff 83 f8 01 0f 86 17 01 00 00
0x0000797aa4d07aa9: 41 8b 45 2c 25 ff 07 00 00 3d c0 04 00 00 41 0f
0x0000797aa4d07ab9: 94 c7 81 fb dd 01 00 00 0f 94 c0 41 08 c7 0f 85
0x0000797aa4d07ac9: f3 00 00 00 49 8b 45 08 4c 8b 40 10 4c 8b 50 08
0x0000797aa4d07ad9: 4c 89 45 c0 4d 85 d2 0f 84 da 00 00 00 41 8b 44
0x0000797aa4d07ae9: 24 34 83 e0 fb 83 f8 02 0f 84 c9 00 00 00 49 8b
0x0000797aa4d07af9: 02 4c 89 d7 4c 89 55 b8 ff 10 4c 8b 45 c0 89 45
0x0000797aa4d07b09: c8 49 8b 00 4c 89 c7 ff 10 4c 8b 55 b8 4c 8b 45
0x0000797aa4d07b19: c0 89 c1 41 f6 42 30 10 74 1d 41 f6 40 30 10 75
0x0000797aa4d07b29: 16 89 d8 83 e0 fd 83 f8 4d 74 0c 81 f9 02 01 00
0x0000797aa4d07b39: 00 0f 85 f8 03 00 00 83 fb 51 0f 84 b7 01 00 00
0x0000797aa4d07b49: 83 fb 52 75 52 41 f7 44 24 34 fb ff ff ff 75 7f
0x0000797aa4d07b59: 83 7d c8 1d 75 79 83 f9 7c 75 74 49 8b 42 08 4c
0x0000797aa4d07b69: 89 45 b8 4c 89 55 c0 48 8b 78 10 48 8b 07 ff 10
Stack slot to memory mapping:
stack at sp + 0 slots: 0x000079789c1b3f98 points into unknown readable memory: 0x0000797aa5226bc0 | c0 6b 22 a5 7a 79 00 00
stack at sp + 1 slots: 0x0000000000068248 is an unknown value
stack at sp + 2 slots: 0x00007978a14d63f0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
stack at sp + 3 slots: 0x00007978a14dcb30 points into unknown readable memory: 0x747361662e6d6f63 | 63 6f 6d 2e 66 61 73 74
stack at sp + 4 slots: 0x00007978a14de3d8 points into unknown readable memory: 0x0000000000007ff5 | f5 7f 00 00 00 00 00 00
stack at sp + 5 slots: 0x0000797a004ebfc0 points into unknown readable memory: 0x0000000000000007 | 07 00 00 00 00 00 00 00
stack at sp + 6 slots: 0x00007978a14dcb30 points into unknown readable memory: 0x747361662e6d6f63 | 63 6f 6d 2e 66 61 73 74
stack at sp + 7 slots: 0x00007978a14d63f0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
In case you missed my comment https://github.com/adoptium/adoptium-support/issues/1191#issuecomment-2705273875
Jackson may be a red herring, I see this too and not using Jackson at all.
Hi all, ideally we need someone to try this out with a fast debug build (https://github.com/adoptium/adoptium-support/issues/1191#issuecomment-2705418499) - are any of you in a position to do that?
Hi all, ideally we need someone to try this out with a fast debug build (#1191 (comment)) - are any of you in a position to do that?
I can reproduce it relatively easily and would be happy to give this a try, but I'm not quite sure what to do. I have downloaded the .debuginfo files and put them into the respective folders bin and lib from my regular temurin build.
Now I can run it and it crashes. But now what? What information can I provide to you? I'm also not sure if I'm doing the right thing with the .debuginfo files or how I could test if they're being picked up?
Turns out we do have fast debug builds - try via our API:
e.g. https://api.adoptium.net/v3/binary/latest/21/ga/linux/x64/debugimage/hotspot/normal/eclipse
@karianna this link gives archive with .debuginfo files, but to me fastdebug build - is build with ./configure --enable-debug - see https://github.com/adoptium/temurin-build/issues/146 , which is not just debug information, but more importantly also
"fastdebug" VM enables internal asserts and verifications
see https://builds.shipilev.net/openjdk-jdk21/
@rethab instead of Temuring build will it be possible for you to try OpenJDK build from the above link https://builds.shipilev.net/openjdk-jdk21/ passing -fastdebug argument to java executable?
instead of Temuring build
Or fastdebug builds provided by Amazon Coretto https://downloads.corretto.aws/#/downloads?build=nightly ?
https://aws.amazon.com/about-aws/whats-new/2023/06/nightly-builds-amazon-corretto/
fast-debug builds, which contain assertions on JVM state and correctness
Thanks @Godin for stepping in! Apologies for sending you down the wrong path @rethab - I'll see if we have infra capacity to actually build the real debug builds on a regular basis, but in the mean time please try one of @Godin's suggestion
I managed to download those builds from shipilev and I believe turning on fastdebug worked by using the script switch-to-fastdebug.sh.
With the release build (switch-to-release.sh) I can reproduce the issue, but once I switch to fastdebug, things are extremely slow and I get various issues with tests such as the spring boot app not even starting up. About 60% of the tests fail with such issues. The crash does not happen though :(
Is there anything else I could try? I can probably build a JDK locally with some flags if that would speed things up or help otherwise..
Hmmm, if the crash doesn't happen with fastdebug then we're going to struggle to get what we need.
You can try building your own: https://github.com/adoptium/adoptium-support/issues/1191#issuecomment-2585588956 and see if that one is any better.