openj9 icon indicating copy to clipboard operation
openj9 copied to clipboard

j9mm.479 * ** ASSERTION FAILED ** at gc_glue_java/ScavengerRootScanner.hpp:108 or MarkingSchemeRootMarker.cpp:53 ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))

Open JasonFengJ9 opened this issue 3 years ago • 16 comments

Failure link

From an internal build job/Test_openjdknext_j9_extended.system_aarch64_linux_Personal/1/tapResults/(cent7-aarch64-6):

openjdk version "19-internal" 2022-09-20
OpenJDK Runtime Environment (build 19-internal-adhoc.jenkins.BuildJDKnextaarch64linuxPersonal)
Eclipse OpenJ9 VM (build exclude19-52f04efbff5, JRE 19 Linux aarch64-64-Bit Compressed References 20220607_35 (JIT enabled, AOT enabled)
OpenJ9   - 52f04efbff5
OMR      - c60867497c6
JCL      - 5ccf02de16a based on jdk-19+25)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

===============================================
Running test SC_Softmx_JitAot_Linux_1 ...
===============================================
SC_Softmx_JitAot_Linux_1 Start Time: Tue Jun  7 13:21:33 2022 Epoch Time (ms): 1654622493065
variation: Mode610
JVM_OPTIONS:  -Xcompressedrefs -Xjit -Xgcpolicy:gencon 

STF 13:21:38.243 - +------ Step 6 - Waiting for jvm1 and jvm2 to finish
STF 13:21:38.243 - | Wait for processes to meet expectations
STF 13:21:38.243 - |   Processes: [jvm1, jvm2]
STF 13:21:38.243 - |
STF 13:21:38.243 - Monitoring processes: jvm1 jvm2
STF 13:21:44.866 - Found dump at: /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/javacore.20220607.132143.14205.0002.txt
STF 13:21:44.867 - Found dump at: /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/core.20220607.132143.14205.0001.dmp
STF 13:21:51.423 - Found dump at: /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/Snap.20220607.132143.14205.0003.trc
STF 13:21:51.425 - **FAILED** Process jvm2 ended with exit code (255) and not the expected exit code/s (0,1)
STF 13:21:51.425 - Monitoring Report Summary:
STF 13:21:51.425 -   o Process jvm1 has crashed unexpectedly
STF 13:21:51.425 -   o Process jvm2 ended with exit code (255) and not the expected exit code/s (0,1)
STF 13:21:51.426 - Killing processes: jvm1 jvm2
STF 13:21:51.426 -   o Process jvm2 pid 14205 is not running
STF 13:21:51.426 -   o Process clean up attempt 1 for jvm1 pid 14204
STF 13:21:51.426 -   o Process jvm1 pid 14204 stop()
STF 13:21:55.427 -   o Process jvm1 pid 14204 killed
**FAILED** at step 6 (Waiting for jvm1 and jvm2 to finish). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/../TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/execute.pl line 211.
STF 13:21:55.472 - **FAILED** execute script failed. Expected return value=0 Actual=1

SC_Softmx_JitAot_Linux_1_FAILED

aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/5.jvm2.stderr 
00000000006BFB00: Object neither in heap nor stack-allocated in thread load-6
00000000006BFB00:       O-Slot=00000000006CAF80
00000000006BFB00:       O-Slot value=0000000000080AF8
00000000006BFB00:       PC=0000FFFF94B91ECC
00000000006BFB00:       framesWalked=3
00000000006BFB00:       arg0EA=00000000006CAFE0
00000000006BFB00:       walkSP=00000000006CADE8
00000000006BFB00:       literals=0000000000000000
00000000006BFB00:       jitInfo=0000FFFF800721C8
00000000006BFB00:       method=0000000000061598 (java/lang/Thread.<init>()V) (JIT)
00000000006BFB00:       stack=00000000006C64F0-00000000006CBD10
17:21:43.490 0x1aca00    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDKnext_aarch64_linux_Personal/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/06/07 13:21:43 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/core.20220607.132143.14205.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/core.20220607.132143.14205.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/javacore.20220607.132143.14205.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/javacore.20220607.132143.14205.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/Snap.20220607.132143.14205.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_linux_Personal_testList_2/aqa-tests/TKG/output_16546195099989/SC_Softmx_JitAot_Linux_1/20220607-132133-SharedClassesWorkloadTest_Softmx_Increase_JitAot/results/Snap.20220607.132143.14205.0003.trc
JVMDUMP013I Processed dump event "traceassert", detail "".

JasonFengJ9 avatar Jun 07 '22 19:06 JasonFengJ9

job/Test_openjdknext_j9_sanity.openjdk_x86-64_mac_Personal/5/consoleFull(mac11x64rt07)

13:14:15  ===============================================
13:14:15  Running test jdk_util_0 ...
13:14:15  ===============================================
13:14:15  jdk_util_0 Start Time: Tue Jun  7 13:14:14 2022 Epoch Time (ms): 1654622054757
13:14:15  "/Users/jenkins/workspace/Test_openjdknext_j9_sanity.openjdk_x86-64_mac_Personal/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/Users/jenkins/workspace/Test_openjdknext_j9_sanity.openjdk_x86-64_mac_Personal/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
13:14:15  JVMSHRC005I No shared class caches available
13:14:15  JVMSHRC005I No shared class caches available
13:14:15  cache cleanup done
13:14:15  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache Mode150

13:21:35  TEST: java/util/ResourceBundle/modules/visibility/VisibilityTest.java

13:21:35  [2022-06-07T17:stderr:
13:21:35  000000001827A000: Object neither in heap nor stack-allocated in thread AgentVMThread
13:21:35  000000001827A000:	O-Slot=0000000018793BC0
13:21:35  000000001827A000:	O-Slot value=0000000017F27AB0
13:21:35  000000001827A000:	PC=000000002DAE36ED
13:21:35  000000001827A000:	framesWalked=5
13:21:35  000000001827A000:	arg0EA=0000000018793C28
13:21:35  000000001827A000:	walkSP=0000000018793AA0
13:21:35  000000001827A000:	literals=0000000017EEF790
13:21:35  000000001827A000:	jitInfo=00007F80BD0A5968
13:21:35  000000001827A000:	method=0000000017EEF890 (java/lang/Thread.<init>(Ljava/lang/Runnable;)V) (JIT)
13:21:35  000000001827A000:	stack=000000001878DAF8-0000000018794B70
13:21:35  17:20:13.394 0x18060900    j9mm.479    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDKnext_x86-64_mac_Personal/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))
13:21:35  JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/06/07 13:20:13 - please wait.

JasonFengJ9 avatar Jun 08 '22 14:06 JasonFengJ9

view/Test_grinder/job/Grinder/24726/tapResults/(ub20x64rt3-2)

===============================================
Running test SharedClasses.SCM23.MultiThread_1 ...
===============================================
SharedClasses.SCM23.MultiThread_1 Start Time: Wed Jun  8 20:49:10 2022 Epoch Time (ms): 1654746550412
variation: Mode610
JVM_OPTIONS:  -Xcompressedrefs -Xjit -Xgcpolicy:gencon 

openjdk version "19-internal" 2022-09-20
OpenJDK Runtime Environment (build 19-internal-adhoc.jenkins.BuildJDKnextx86-64linuxPersonal)
Eclipse OpenJ9 VM (build exclude19-622b1ac1f26, JRE 19 Linux amd64-64-Bit Compressed References 20220608_1018 (JIT enabled, AOT enabled)
OpenJ9   - 622b1ac1f26
OMR      - c859f0d17d0
JCL      - 4cb748300c8 based on jdk-19+25)

MT2 20:58:21 >> Loaded 15000 classes...
MT5 stderr 0000000000016F00: Object neither in heap nor stack-allocated in thread main
MT5 stderr 0000000000016F00:	O-Slot=0000000000115578
MT5 stderr 0000000000016F00:	O-Slot value=00000000000B20B0
MT5 stderr 0000000000016F00:	PC=00007F3711F81CC2
MT5 stderr 0000000000016F00:	framesWalked=1
MT5 stderr 0000000000016F00:	arg0EA=0000000000115620
MT5 stderr 0000000000016F00:	walkSP=00000000001154E8
MT5 stderr 0000000000016F00:	literals=0000000000000000
MT5 stderr 0000000000016F00:	jitInfo=00007F3711614BF8
MT5 stderr 0000000000016F00:	method=00000000001FD830 (net/openj9/test/sc/LoaderSlaveMultiThread.run(Ljava/lang/String;I)V) (JIT)
MT5 stderr 0000000000016F00:	stack=000000000010E010-0000000000115730
MT5 stderr 03:58:21.356 0x1c3200    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDKnext_x86-64_linux_Personal/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))
MT5 stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/06/08 20:58:21 - please wait.

STF 22:54:17.293 - Heartbeat: Process MT5 is still running
STF 22:57:19.507 - **FAILED** Process MT5 has timed out
STF 22:57:19.519 - Collecting dumps for: MT5

STF 22:59:32.959 - Monitoring Report Summary:
STF 22:59:32.959 -   o Process SCC has crashed unexpectedly
STF 22:59:32.960 - Killing processes: SCC
STF 22:59:32.960 -   o Process SCC pid 3087881 is not running
**FAILED** at step 8 (Destroy cache). Expected return value=0 Actual=1 at /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_16547269554530/SharedClasses.SCM23.MultiThread_1/20220608-204910-SharedClasses/tearDown.pl line 67.
STF 22:59:32.969 - **FAILED** teardown script failed. Expected return value=0 Actual=1
STF 22:59:32.969 - 
STF 22:59:32.969 - =====================   R E S U L T S   =====================
STF 22:59:32.973 - Stage results:
STF 22:59:32.981 -   setUp:     pass
STF 22:59:32.984 -   execute:  *fail*
STF 22:59:32.984 -   teardown:  pass
STF 22:59:32.985 - 
STF 22:59:32.985 - Overall result: **FAILED**

SharedClasses.SCM23.MultiThread_1_FAILED

view/Test_grinder/job/Grinder/24722/consoleFull

01:39:06  ===============================================
01:39:06  Running test SharedClasses.SCM23.MultiThreadMultiCL_0 ...
01:39:06  ===============================================
01:39:06  SharedClasses.SCM23.MultiThreadMultiCL_0 Start Time: Wed Jun  8 22:39:06 2022 Epoch Time (ms): 1654753146392
01:39:06  "/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
01:39:06  JVMSHRC005I No shared class caches available
01:39:06  JVMSHRC005I No shared class caches available
01:39:06  cache cleanup done
01:39:06  variation: Mode110
01:39:06  JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

01:39:13  openjdk version "19-internal" 2022-09-20
01:39:13  OpenJDK Runtime Environment (build 19-internal-adhoc.jenkins.BuildJDKnextx86-64linuxPersonal)
01:39:13  Eclipse OpenJ9 VM (build exclude19-622b1ac1f26, JRE 19 Linux amd64-64-Bit Compressed References 20220608_1018 (JIT enabled, AOT enabled)
01:39:13  OpenJ9   - 622b1ac1f26
01:39:13  OMR      - c859f0d17d0
01:39:13  JCL      - 4cb748300c8 based on jdk-19+25)

01:59:40  MTM5 22:59:39 >> --> Threads 9000 to 9120 <--
01:59:40  MTM2 stderr 00007F5A800CAA00: Object neither in heap nor stack-allocated in thread main
01:59:40  MTM2 stderr 00007F5A800CAA00:	O-Slot=00007F5A8038BA30
01:59:40  MTM2 stderr 00007F5A800CAA00:	O-Slot value=00007F5A802FC5B0
01:59:40  MTM2 stderr 00007F5A800CAA00:	PC=00007F5A26BF0DC8
01:59:40  MTM2 stderr 00007F5A800CAA00:	framesWalked=1
01:59:40  MTM2 stderr 00007F5A800CAA00:	arg0EA=00007F5A8038BAD0
01:59:40  MTM2 stderr 00007F5A800CAA00:	walkSP=00007F5A8038B998
01:59:40  MTM2 stderr 00007F5A800CAA00:	literals=0000000000000000
01:59:40  MTM2 stderr 00007F5A800CAA00:	jitInfo=00007F5991136EF8
01:59:40  MTM2 stderr 00007F5A800CAA00:	method=00007F5A805F1610 (net/openj9/test/sc/LoaderSlaveMultiThreadMultiCL.run(Ljava/lang/String;I)V) (JIT)
01:59:40  MTM2 stderr 00007F5A800CAA00:	stack=00007F5A803844B0-00007F5A8038BBD0
01:59:40  MTM2 stderr 05:59:39.620 0x7f59d8003c00    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDKnext_x86-64_linux_Personal/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))
01:59:40  MTM2 stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/06/08 22:59:39 - please wait.

04:00:02  SharedClasses.SCM23.MultiThreadMultiCL_0_FAILED

JasonFengJ9 avatar Jun 09 '22 12:06 JasonFengJ9

Didn't reproduce this 1x in https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/1038 Trying again 3x x 3 https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/1042

pshipton avatar Jun 23 '22 21:06 pshipton

The grinder failed, but the results were purged. Trying another 1x (by accident) https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/1070/

pshipton avatar Jun 27 '22 20:06 pshipton

The failure is recreated. @dmitripivkine pls take a look.

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Grinder/1070/system_test_output.tar.gz

pshipton avatar Jun 27 '22 22:06 pshipton

For recreated failure: There is bad O-slot 0x5cb1e0 in JIT frame for method=0x69718 (java/lang/Thread.<init>()V) (JIT) in java stack for thread 0x5bc600 contains 0xa29b0 not in heap or thread stack ranges:

20:08:22.424668000 *0x1B4C00 j9mm.481             Exception  * Object neither in heap nor stack-allocated in thread load-2
20:08:22.424672000  0x1B4C00 j9mm.482             Exception  * O-Slot=0x5cb1e0
20:08:22.424675000  0x1B4C00 j9mm.483             Exception  * O-Slot value=0xa29b0
20:08:22.424677000  0x1B4C00 j9mm.484             Exception  * PC=0xffff9cfc0cb0
20:08:22.424680000  0x1B4C00 j9mm.485             Exception  * framesWalked=5
20:08:22.424682000  0x1B4C00 j9mm.486             Exception  * arg0EA=0x5cb240
20:08:22.424684000  0x1B4C00 j9mm.487             Exception  * walkSP=0x5caf50
20:08:22.424686000  0x1B4C00 j9mm.488             Exception  * literals=0x8
20:08:22.424688000  0x1B4C00 j9mm.489             Exception  * jitInfo=0xffff94b048b8
20:08:22.424692000  0x1B4C00 j9mm.490             Exception  * method=0x69718 (java/lang/Thread.<init>()V) (JIT)
20:08:22.424695000  0x1B4C00 j9mm.493             Exception  * stack=0x5c66b0-0x5cbef0
20:08:22.424695000  0x1B4C00 j9mm.491             Exit       *<StackSlotValidator::reportStackSlot
20:08:22.424696000  0x1B4C00 j9mm.479             Assert     * ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_aarch64_linux_Nightly/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))

<5bc600> JIT frame: bp = 0x00000000005CB238, pc = 0x0000FFFF9CFC0CB0, unwindSP = 0x00000000005CB170, cp = 0x0000000000067D90, arg0EA = 0x00000000005CB240, jitInfo = 0x0000FFFF94B048B8
<5bc600> 	Method: java/lang/Thread.<init>()V !j9method 0x0000000000069718
<5bc600> 	Bytecode index = 9, inlineDepth = 0, PC offset = 0x00000000000001D8
<5bc600> 	stackMap=0x0000FFFF94B04A58, slots=I16(0x0001) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0001), localBaseOffset=I16(0xFFA0)
<5bc600> 	Described JIT args starting at 0x00000000005CB240 for U16(0x0001) slots
<5bc600> 		O-Slot: : a0[0x00000000005CB240] = 0x00000000FFC1F700
<5bc600> 	Described JIT temps starting at 0x00000000005CB1D8 for IDATA(0x000000000000000C) slots
<5bc600> 		O-Slot: : t11[0x00000000005CB1D8] = 0x0000000088FADFB8
<5bc600> 		O-Slot: : t10[0x00000000005CB1E0] = 0x00000000000A29B0 <----- from "Classes"
<5bc600> 		O-Slot: : t9[0x00000000005CB1E8] = 0x0000000088F65AE0
<5bc600> 		O-Slot: : t8[0x00000000005CB1F0] = 0x00000000FF827910
<5bc600> 		O-Slot: : t7[0x00000000005CB1F8] = 0x00000000FF8278C8
<5bc600> 		O-Slot: : t6[0x00000000005CB200] = 0x00000000FF8278C8
<5bc600> 		O-Slot: : t5[0x00000000005CB208] = 0x00000000FF8278C8
<5bc600> 		O-Slot: : t4[0x00000000005CB210] = 0x00000000FF827910
<5bc600> 		O-Slot: : t3[0x00000000005CB218] = 0x00000000FF8278C8
<5bc600> 		O-Slot: : t2[0x00000000005CB220] = 0x00000000FF8278C8
<5bc600> 		O-Slot: : t1[0x00000000005CB228] = 0x00000000FF8278C8
<5bc600> 		O-Slot: : t0[0x00000000005CB230] = 0x00000000FF827910
<5bc600> 	JIT-RegisterMap = UDATA(0x0000000000000000)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CB1B0] = UDATA(0x00000000FF827738) (jit_r21)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CB098] = UDATA(0x0000000000000000) (jit_r22)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CAFB8] = UDATA(0x0000000000000000) (jit_r23)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CAFC0] = UDATA(0x0000000000059600) (jit_r24)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CAFC8] = UDATA(0x0000000000064A00) (jit_r25)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CAFD0] = UDATA(0x00000000FFCC7978) (jit_r26)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CAFD8] = UDATA(0x0000FFFFBDE6DBF2) (jit_r27)
<5bc600> 	JIT-Frame-RegisterMap[0x00000000005CAFE0] = UDATA(0x0000FFFFBDE6DB22) (jit_r28)

Actually bogus value 0xa29b0 is an address of ramStatics for !j9class 0xA6200 java/lang/Thread$ThreadNumbering:

> !j9statics 0x00000000000A6200
Static fields in java/lang/Thread$ThreadNumbering:
	0x00000000000A29B0 U Ljdk/internal/misc/Unsafe; (!j9romstaticfieldshape 0x0000FFFF957ABDB0) = !j9object 0x0000000088FADFB8
	0x00000000000A29C0 NEXT J (!j9romstaticfieldshape 0x0000FFFF957ABDBC) = 0x0000000000000009 (9)
	0x00000000000A29B8 next I (!j9romstaticfieldshape 0x0000FFFF957ABDC8) = 0x00000019 (25)

I wondering can it be missed indirection and intention was to get object pointer from ramStatics @0xdaryl FYI

dmitripivkine avatar Jun 28 '22 13:06 dmitripivkine

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_OpenJDK19/4 jdk_util_0 java/util/concurrent/FutureTask/ExplicitSet.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_OpenJDK19/4/openjdk_test_output.tar.gz

15:32:43  stderr:
15:32:43  0000000000543100: Object neither in heap nor stack-allocated in thread AgentVMThread
15:32:43  0000000000543100:	O-Slot=00000000006F4708
15:32:43  0000000000543100:	O-Slot value=00000000000A29B0
15:32:43  0000000000543100:	PC=0000FFFF8DCAC888
15:32:43  0000000000543100:	framesWalked=1
15:32:43  0000000000543100:	arg0EA=00000000006F4728
15:32:43  0000000000543100:	walkSP=00000000006F4698
15:32:43  0000000000543100:	literals=0000000000000000
15:32:43  0000000000543100:	jitInfo=0000FFFF6D86A238
15:32:43  0000000000543100:	method=00000000000696D8 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
15:32:43  0000000000543100:	stack=00000000006EF288-00000000006F4AE0
15:32:43  19:32:32.515 0x543100    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_aarch64_linux_OpenJDK19/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))

pshipton avatar Jun 30 '22 15:06 pshipton

job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_testList_0/8/(paix819)

openjdk version "19" 2022-09-20
IBM Semeru Runtime Open Edition 19+29 (build 19+29)
Eclipse OpenJ9 VM 19+29 (build master-6b4a2db09, JRE 19 AIX ppc64-64-Bit Compressed References 20220919_13 (JIT enabled, AOT enabled)
OpenJ9   - 6b4a2db09
OMR      - ff6a49823
JCL      - 403c2088e2 based on jdk-19+29)
[2022-07-09T17:17:19.679Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
[2022-07-09T17:17:19.679Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops 

[2022-07-09T17:40:41.124Z] TEST: java/lang/RuntimeTests/exec/SleepyCat.java

[2022-07-09T17:40:41.125Z] stderr:
[2022-07-09T17:40:41.125Z] 0000010010433200: Invalid class pointer in thread main
[2022-07-09T17:40:41.125Z] 0000010010433200:	O-Slot=000001001009C648
[2022-07-09T17:40:41.125Z] 0000010010433200:	O-Slot value=0A0000001FFB61F0
[2022-07-09T17:40:41.125Z] 0000010010433200:	PC=0000010010777C78
[2022-07-09T17:40:41.125Z] 0000010010433200:	framesWalked=9
[2022-07-09T17:40:41.125Z] 0000010010433200:	arg0EA=00000100220B1908
[2022-07-09T17:40:41.125Z] 0000010010433200:	walkSP=00000100220B1898
[2022-07-09T17:40:41.125Z] 0000010010433200:	literals=0000000000000010
[2022-07-09T17:40:41.125Z] 0000010010433200:	jitInfo=0000010021BD7498
[2022-07-09T17:40:41.125Z] 0000010010433200:	method=0000010023610B48 (java/lang/Thread$$Lambda$66/0x0000000000000000.test(Ljava/lang/Object;)Z) (JIT)
[2022-07-09T17:40:41.125Z] 0000010010433200:	stack=00000100220AC810-00000100220B20C0
[2022-07-09T17:40:41.125Z] 17:40:33.331 0x10022c41600    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-aix-ppc64-openj9/workspace/build/src/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:48: ((MM_StackSlotValidator(0, object, stackLocation, walkState).validate(_env)))
[2022-07-09T17:40:41.125Z] JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/07/09 13:40:33 - please wait.

[2022-07-09T17:51:11.339Z] jdk_lang_1_FAILED

JasonFengJ9 avatar Jul 11 '22 21:07 JasonFengJ9

An internal build(ubu22s390x-rt3-1)

java version "11.0.16" 2022-07-19
IBM Semeru Runtime Certified Edition 11.0.16.0-rc1a (build 11.0.16+8)
Eclipse OpenJ9 VM 11.0.16.0-rc1a (build openj9-0.33.0-rc1a, JRE 11 Linux s390x-64-Bit Compressed References 20220721_339 (JIT enabled, AOT enabled)
OpenJ9   - 04a55b45b
OMR      - b58aa2708
JCL      - 78837d3d66 based on jdk-11.0.16+8)

[2022-07-22T03:41:01.438Z] variation: Mode650
[2022-07-22T03:41:01.438Z] JVM_OPTIONS:  -XX:-UseCompressedOops 

[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00: Invalid class pointer in stack allocated object in thread Thread-66
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	O-Slot=000003FE94008D30
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	O-Slot value=000003FE94008BC8
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	PC=000003FFA04D8E50
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	framesWalked=3
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	arg0EA=000003FE94008D38
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	walkSP=000003FE94008D10
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	literals=000003FFA04EC498
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	jitInfo=0000000000000000
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	method=000003FFA04EC498 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0A2A00:	stack=000003FE94003550-000003FE94008D90
[2022-07-22T03:42:13.706Z] LT  stderr 03:42:06.197 0x3fe9c087500    j9mm.479    *   ** ASSERTION FAILED ** at ../../../../../../openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/07/22 03:42:06 - please wait.
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/core.20220722.034206.186448.0001.dmp' in response to an event
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00: Invalid class pointer in stack allocated object in thread Thread-67
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	O-Slot=000003FE840206B0
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	O-Slot value=000003FE84020548
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	PC=000003FFA04D8E77
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	framesWalked=3
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	arg0EA=000003FE840206B8
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	walkSP=000003FE84020690
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	literals=000003FFA04EC498
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	jitInfo=0000000000000000
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	method=000003FFA04EC498 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C08DE00:	stack=000003FE8401AED0-000003FE84020710
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00: Invalid class pointer in stack allocated object in thread Thread-68
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	O-Slot=000003FE80071750
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	O-Slot value=000003FE800715E8
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	PC=000003FFA04D8E50
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	framesWalked=3
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	arg0EA=000003FE80071758
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	walkSP=000003FE80071730
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	literals=000003FFA04EC498
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	jitInfo=0000000000000000
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	method=000003FFA04EC498 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
[2022-07-22T03:42:13.706Z] LT  stderr 000003FE9C0B0D00:	stack=000003FE8006BF70-000003FE800717B0
[2022-07-22T03:42:13.706Z] STF 03:42:09.909 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/javacore.20220722.034206.186448.0002.txt
[2022-07-22T03:42:13.706Z] STF 03:42:09.910 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/core.20220722.034206.186448.0001.dmp
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/core.20220722.034206.186448.0001.dmp
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/javacore.20220722.034206.186448.0002.txt' in response to an event
[2022-07-22T03:42:13.706Z] AG  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/javacore.20220722.034206.186448.0002.txt
[2022-07-22T03:42:13.706Z] AG  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/core.20220722.034206.186448.0001.dmp
[2022-07-22T03:42:13.706Z] AG  stderr Exception in thread "main" java.net.SocketException: Broken pipe (Write failed)
[2022-07-22T03:42:13.706Z] AG  stderr 	at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
[2022-07-22T03:42:13.706Z] AG  stderr 	at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:129)
[2022-07-22T03:42:13.706Z] AG  stderr 	at java.base/openj9.internal.tools.attach.target.AttachmentConnection.streamSend(AttachmentConnection.java:105)
[2022-07-22T03:42:13.706Z] AG  stderr 	at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.detach(OpenJ9VirtualMachine.java:206)
[2022-07-22T03:42:13.706Z] AG  stderr 	at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:85)
[2022-07-22T03:42:13.706Z] STF 03:42:12.428 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/Snap.20220722.034206.186448.0003.trc
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/javacore.20220722.034206.186448.0002.txt
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/Snap.20220722.034206.186448.0003.trc' in response to an event
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/Snap.20220722.034206.186448.0003.trc
[2022-07-22T03:42:13.706Z] LT  stderr JVMDUMP013I Processed dump event "traceassert", detail "".
[2022-07-22T03:42:13.706Z] STF 03:42:12.430 - **FAILED** Process LT  has ended unexpectedly
[2022-07-22T03:42:13.706Z] AG  stderr Snap file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/results/Snap.20220722.034206.186448.0003.trc
[2022-07-22T03:42:13.706Z] STF 03:42:12.431 - Monitoring Report Summary:
[2022-07-22T03:42:13.706Z] STF 03:42:12.431 -   o Process AG  has crashed unexpectedly
[2022-07-22T03:42:13.706Z] STF 03:42:12.431 -   o Process LT  has ended unexpectedly
[2022-07-22T03:42:13.706Z] STF 03:42:12.431 - Killing processes: AG  LT 
[2022-07-22T03:42:13.706Z] STF 03:42:12.431 -   o Process clean up attempt 1 for AG  pid 186451
[2022-07-22T03:42:13.706Z] STF 03:42:12.431 -   o Process AG  pid 186451 stop()
[2022-07-22T03:42:14.439Z] STF 03:42:13.433 -   o Process AG  pid 186451 killed
[2022-07-22T03:42:14.439Z] STF 03:42:13.433 -   o Process LT  pid 186448 is not running
[2022-07-22T03:42:14.440Z] **FAILED** at step 3 (Wait for Agent to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/../TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/execute.pl line 157.
[2022-07-22T03:42:14.440Z] STF 03:42:13.828 - **FAILED** execute script failed. Expected return value=0 Actual=1
[2022-07-22T03:42:14.440Z] STF 03:42:13.828 - 
[2022-07-22T03:42:14.440Z] STF 03:42:13.828 - ====================   T E A R D O W N   ====================
[2022-07-22T03:42:14.440Z] STF 03:42:13.828 - Running teardown: perl /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/../TKG/output_16584587027484/HCRLateAttachWorkload_previewEnabled_1/20220722-034101-HCRLateAttachWorkload/tearDown.pl
[2022-07-22T03:42:14.440Z] STF 03:42:13.936 - TEARDOWN stage completed
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 - 
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 - =====================   R E S U L T S   =====================
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 - Stage results:
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 -   setUp:     pass
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 -   execute:  *fail*
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 -   teardown:  pass
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 - 
[2022-07-22T03:42:14.440Z] STF 03:42:13.945 - Overall result: **FAILED**
[2022-07-22T03:42:14.440Z] 
[2022-07-22T03:42:14.440Z] HCRLateAttachWorkload_previewEnabled_1_FAILED

50x internal grinder - passed

Same assertion as https://github.com/eclipse-openj9/openj9/issues/15476

JasonFengJ9 avatar Jul 25 '22 14:07 JasonFengJ9

An internal build(p10sles021)

[2022-08-28T09:20:36.685Z] variation: Mode112
[2022-08-28T09:20:36.685Z] JVM_OPTIONS:  -Xgcpolicy:gencon -Xjit:count=0 -Xnocompressedrefs 

[2022-08-28T09:20:38.733Z] openjdk version "19" 2022-09-20
[2022-08-28T09:20:38.733Z] IBM Semeru Runtime Open Edition 19+36 (build 19+36)
[2022-08-28T09:20:38.733Z] Eclipse OpenJ9 VM 19+36 (build master-c14fe746e, JRE 19 Linux ppc64le-64-Bit Compressed References 20220919_32 (JIT enabled, AOT enabled)
[2022-08-28T09:20:38.733Z] OpenJ9   - c14fe746e
[2022-08-28T09:20:38.733Z] OMR      - 938f0686f
[2022-08-28T09:20:38.733Z] JCL      - f52075642c based on jdk-19+36)

[2022-08-28T09:21:41.693Z] LT  05:21:36.617 - Completed 14.9%. Number of tests started=4073 (+2522)
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900: Object neither in heap nor stack-allocated in thread load-27
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	O-Slot=00007FFDCC05E0E0
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	O-Slot value=00007FFFA4426FB0
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	PC=00007FFF0EF9FA14
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	framesWalked=2
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	arg0EA=00007FFDCC05E0F8
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	walkSP=00007FFDCC05E018
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	literals=0000000000000000
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	jitInfo=00007FFE764C8C78
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	method=00007FFFA423F860 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
[2022-08-28T09:21:44.958Z] LT  stderr 00007FFFA4C83900:	stack=00007FFDCC024FA8-00007FFDCC05EB60
[2022-08-28T09:21:44.958Z] LT  stderr 09:21:43.780 0x7ffe68002400    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))
[2022-08-28T09:21:44.958Z] LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/08/28 05:21:43 - please wait.

[2022-08-28T09:21:46.868Z] LambdaLoadTest_special_J9_5m_4_FAILED

Same as https://github.com/eclipse-openj9/openj9/issues/13369 which was marked as dup of this.

JasonFengJ9 avatar Sep 01 '22 16:09 JasonFengJ9

An internal build(ub18-aarch64-1)

[2022-08-28T00:57:54.299Z] variation: Mode110
[2022-08-28T00:57:54.299Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

[2022-08-28T00:57:59.706Z] openjdk version "19" 2022-09-20
[2022-08-28T00:57:59.706Z] IBM Semeru Runtime Open Edition 19+36 (build 19+36)
[2022-08-28T00:57:59.706Z] Eclipse OpenJ9 VM 19+36 (build master-c14fe746e, JRE 19 Linux aarch64-64-Bit Compressed References 20220920_32 (JIT enabled, AOT enabled)
[2022-08-28T00:57:59.706Z] OpenJ9   - c14fe746e
[2022-08-28T00:57:59.706Z] OMR      - 938f0686f
[2022-08-28T00:57:59.706Z] JCL      - f52075642c based on jdk-19+36)

[2022-08-28T00:58:46.461Z] LT  20:58:43.519 - Completed 13.4%. Number of tests started=1366 (+643)
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00: Object neither in heap nor stack-allocated in thread load-1
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	O-Slot=0000FFFEA01B2168
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	O-Slot value=0000FFFFB838B7B8
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	PC=0000FFFF1E56EF24
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	framesWalked=1
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	arg0EA=0000FFFEA01B2188
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	walkSP=0000FFFEA01B20F8
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	literals=0000000000000000
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	jitInfo=0000FFFEE4715D60
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	method=0000FFFFB819F6C0 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
[2022-08-28T00:58:46.461Z] LT  stderr 0000FFFFB8B1AC00:	stack=0000FFFEA0148CF8-0000FFFEA01B2B00
[2022-08-28T00:58:46.461Z] LT  stderr 00:58:43.988 0xfffec0002400    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-aarch64-openj9/workspace/build/src/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:108: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))
[2022-08-28T00:58:46.461Z] LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/08/27 20:58:43 - please wait.

[2022-08-28T00:58:49.987Z] LambdaLoadTest_J9_5m_0_FAILED

JasonFengJ9 avatar Sep 01 '22 16:09 JasonFengJ9

Internal build (p10rhel053) LambdaLoadTest_special_J9_5m_1 -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs

[2022-08-30T09:47:36.676Z] openjdk version "19-beta" 2022-09-20
[2022-08-30T09:47:36.676Z] IBM Semeru Runtime Open Edition 19+36-202208300535 (build 19-beta+36-202208300535)
[2022-08-30T09:47:36.676Z] Eclipse OpenJ9 VM 19+36-202208300535 (build master-e53d18b81, JRE 19 Linux ppc64le-64-Bit Compressed References 20220830_33 (JIT enabled, AOT enabled)
[2022-08-30T09:47:36.676Z] OpenJ9   - e53d18b81
[2022-08-30T09:47:36.676Z] OMR      - 938f0686f
[2022-08-30T09:47:36.676Z] JCL      - f52075642c based on jdk-19+36)

[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400: Object neither in heap nor stack-allocated in thread load-24
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	O-Slot=00007FFDD8048EC0
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	O-Slot value=00007FFF9440FCB0
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	PC=00007FFF00D415E4
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	framesWalked=1
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	arg0EA=00007FFDD8048ED8
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	walkSP=00007FFDD8048E58
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	literals=0000000000000000
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	jitInfo=00007FFEC8267878
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	method=00007FFF9422E940 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
[2022-08-30T09:48:24.112Z] LT  stderr 00007FFF94C6A400:	stack=00007FFDD8027F78-00007FFDD80499A0
[2022-08-30T09:48:24.112Z] LT  stderr 09:48:19.527 0x7ffe58002400    j9mm.479    *   ** ASSERTION FAILED ** at ../../../../../../openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))

pshipton avatar Sep 08 '22 00:09 pshipton

Internal build(macaarch64rt1)

[2022-09-11T12:25:18.326Z] variation: Mode101
[2022-09-11T12:25:18.326Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:optthruput -Xnocompressedrefs 

[2022-09-11T12:25:19.615Z] openjdk version "19-ea" 2022-09-20
[2022-09-11T12:25:19.615Z] IBM Semeru Runtime Open Edition 19+36 (build 19-ea+36)
[2022-09-11T12:25:19.615Z] Eclipse OpenJ9 VM 19+36 (build master-b8752ec8b, JRE 19 Mac OS X aarch64-64-Bit 20220920_37 (JIT enabled, AOT enabled)
[2022-09-11T12:25:19.615Z] OpenJ9   - b8752ec8b
[2022-09-11T12:25:19.615Z] OMR      - 647678f93
[2022-09-11T12:25:19.615Z] JCL      - 034a8c8f12 based on jdk-19+36)

[2022-09-11T12:28:21.399Z] LT  08:28:20.377 - Completed 60.0%. Number of tests started=11882 (+1241)
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00: Object neither in heap nor stack-allocated in thread load-3
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	O-Slot=0000000158521700
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	O-Slot value=00000001517FF3B8
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	PC=000000010FB57B9C
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	framesWalked=1
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	arg0EA=0000000158521718
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	walkSP=0000000158521698
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	literals=0000000000000000
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	jitInfo=00000001602C64A8
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	method=00000001515D03F0 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
[2022-09-11T12:28:38.365Z] LT  stderr 00000001490C4F00:	stack=00000001584B8048-0000000158522090
[2022-09-11T12:28:38.365Z] LT  stderr 12:28:36.322 0x149171700    j9mm.479    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-mac-aarch64-openj9/workspace/build/src/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))
[2022-09-11T12:28:38.365Z] LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/09/11 08:28:36 - please wait.

[2022-09-11T12:28:54.857Z] LambdaLoadTest_special_J9_5m_0_FAILED

JasonFengJ9 avatar Sep 12 '22 21:09 JasonFengJ9

Another occurrence https://github.com/eclipse-openj9/openj9/issues/15885

pshipton avatar Sep 14 '22 16:09 pshipton

An internal build(p10sles021)

openjdk version "19-beta" 2022-09-20
IBM Semeru Runtime Open Edition 19+36-202209200534 (build 19-beta+36-202209200534)
Eclipse OpenJ9 VM 19+36-202209200534 (build master-2184d058b, JRE 19 Linux ppc64le-64-Bit Compressed References 20220920_43 (JIT enabled, AOT enabled)
OpenJ9   - 2184d058b
OMR      - b1be89cf5
JCL      - b56bf70733 based on jdk-19+36)
[2022-09-20T08:52:21.700Z] variation: Mode121
[2022-09-20T08:52:21.700Z] JVM_OPTIONS:  -Xgcpolicy:optavgpause -Xjit:count=0 -Xnocompressedrefs 

[2022-09-20T08:53:05.531Z] LT  stderr 08:53:04.123 0x7ffe78002400    j9mm.479    *   ** ASSERTION FAILED ** at ../../../../../../openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))

JasonFengJ9 avatar Sep 20 '22 22:09 JasonFengJ9

LT  stderr 00007FFF84C6E300: Object neither in heap nor stack-allocated in thread load-29
LT  stderr 00007FFF84C6E300:	O-Slot=00007FFDA80A2650
LT  stderr 00007FFF84C6E300:	O-Slot value=00007FFF843F33B0
LT  stderr 00007FFF84C6E300:	PC=00007FFEEFE20014
LT  stderr 00007FFF84C6E300:	framesWalked=2
LT  stderr 00007FFF84C6E300:	arg0EA=00007FFDA80A2668
LT  stderr 00007FFF84C6E300:	walkSP=00007FFDA80A2568
LT  stderr 00007FFF84C6E300:	literals=0000000000000000
LT  stderr 00007FFF84C6E300:	jitInfo=00007FFEB418C138
LT  stderr 00007FFF84C6E300:	method=00007FFF8420EFE0 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
LT  stderr 00007FFF84C6E300:	stack=00007FFDA80715F8-00007FFDA80A3150

dmitripivkine avatar Sep 20 '22 23:09 dmitripivkine

Internal build(macaarch64rt1)

openjdk version "19-ea" 2022-09-20
IBM Semeru Runtime Open Edition 19+36-202209220534 (build 19-ea+36-202209220534)
Eclipse OpenJ9 VM 19+36-202209220534 (build master-5a2eb17e7, JRE 19 Mac OS X aarch64-64-Bit 20220922_43 (JIT enabled, AOT enabled)
OpenJ9   - 5a2eb17e7
OMR      - b1be89cf5
JCL      - 4efd563f9c based on jdk-19+36)

[2022-09-22T12:24:29.867Z] variation: Mode113
[2022-09-22T12:24:29.867Z] JVM_OPTIONS: -Xdump:system:events=user

[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500: Object neither in heap nor stack-allocated in thread load-3
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	O-Slot=00000001705E9108
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	O-Slot value=00000001482EF6B8
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	PC=0000000114D65D1C
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	framesWalked=1
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	arg0EA=00000001705E9128
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	walkSP=00000001705E9098
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	literals=0000000000000000
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	jitInfo=000000014805C5E8
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	method=00000001395883F0 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
[2022-09-22T12:25:53.759Z] LT  stderr 00000001418DF500:	stack=0000000170580048-00000001705E9DF0
[2022-09-22T12:25:53.759Z] LT  stderr 12:25:52.870 0x131034d00    j9mm.479    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-mac-aarch64-openj9/workspace/build/src/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:109: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))

[2022-09-22T12:26:14.214Z] LambdaLoadTest_special_J9_5m_5_FAILED

Same error as this issue as per https://github.com/eclipse-openj9/openj9/issues/15885

JasonFengJ9 avatar Sep 23 '22 20:09 JasonFengJ9

another two failures in https://github.com/eclipse-openj9/openj9/issues/15952

In both cases bogus value is j9class-> ramStatics for java/lang/Thread$ThreadNumbering

first core:

LT  stderr 0000000000404100: O-Slot=0000000000AFCA18
LT  stderr 0000000000404100: O-Slot value=0000000000149FB0 <-------

> !allclasses | grep java/lang/Thread$ThreadNumbering
0x000000000015E100	0x0000000000000328	0x0000FFFF840C3AC8	0x0000FFFF2D6C0D08	0	java/lang/Thread$ThreadNumbering
0x0000FFFF2D6C0D08	0x00000220	0x0000FFFF840C3AC8	0x00000020	0x0C804000	java/lang/Thread$ThreadNumbering
> !j9class 0x000000000015E100 | grep ramStatics
	0x48: UDATA* ramStatics = !j9x 0x0000000000149FB0 <-------

second core:

DBLT stderr 00000000305C2200:	O-Slot=0000000030604D30
DBLT stderr 00000000305C2200:	O-Slot value=00000000301489B0 <-------

> !allclasses | grep java/lang/Thread$ThreadNumbering
0x0000000030148A00	0x0000000000000328	0x00000100103F39F8	0x0A000100091ECBE0	0	java/lang/Thread$ThreadNumbering
0x0A000100091ECBE0	0x00000220	0x00000100103F39F8	0x00000020	0x0C804000	java/lang/Thread$ThreadNumbering
> !j9class 0x0000000030148A00 | grep ramStatics
	0x48: UDATA* ramStatics = !j9x 0x00000000301489B0 <-----

dmitripivkine avatar Sep 26 '22 13:09 dmitripivkine

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_ppc64_aix_Nightly_testList_1/13 DBBLoadTest_5m_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.system_ppc64_aix_Nightly_testList_1/13/system_test_output.tar.gz

DBLT 06:28:27.821 - Completed 6.7%. Number of tests started=175
DBLT stderr 00000000305CEF00: Object neither in heap nor stack-allocated in thread load-13
DBLT stderr 00000000305CEF00:	O-Slot=00000000305D4D30
DBLT stderr 00000000305CEF00:	O-Slot value=00000000301489B0
DBLT stderr 00000000305CEF00:	PC=0000010010DBE11C
DBLT stderr 00000000305CEF00:	framesWalked=1
DBLT stderr 00000000305CEF00:	arg0EA=00000000305D4D48
DBLT stderr 00000000305CEF00:	walkSP=00000000305D4CC0
DBLT stderr 00000000305CEF00:	literals=0000000000000000
DBLT stderr 00000000305CEF00:	jitInfo=0000010021AD49F8
DBLT stderr 00000000305CEF00:	method=0000000030060CE0 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
DBLT stderr 00000000305CEF00:	stack=00000000305CFB40-00000000305D5350
DBLT stderr 06:28:43.909 0x3014fa00    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_ppc64_aix_Nightly/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))

pshipton avatar Sep 26 '22 20:09 pshipton

The same failure is mentioned in https://github.com/eclipse-openj9/openj9/issues/15170#issuecomment-1256825887

dmitripivkine avatar Sep 27 '22 12:09 dmitripivkine

Check core from https://github.com/eclipse-openj9/openj9/issues/15251#issuecomment-1258605223 - yes, there is the same problem:

DBLT stderr 00000000305CEF00:	O-Slot=00000000305D4D30
DBLT stderr 00000000305CEF00:	O-Slot value=00000000301489B0 <-------

> !allclasses | grep java/lang/Thread$ThreadNumbering
0x0000000030148A00	0x0000000000000328	0x00000100103F4098	0x0A000100091ECBD8	0	java/lang/Thread$ThreadNumbering
0x0A000100091ECBD8	0x00000220	0x00000100103F4098	0x00000020	0x0C804000	java/lang/Thread$ThreadNumbering
> !j9class 0x0000000030148A00 | grep ramStatics
	0x48: UDATA* ramStatics = !j9x 0x00000000301489B0 <-------

dmitripivkine avatar Sep 27 '22 12:09 dmitripivkine

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_1/15/ LambdaLoadTest_CS_5m_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_1/15/system_test_output.tar.gz

LT  02:51:52.520 - Completed 20.0%. Number of tests started=3292 (+1416)
LT  stderr 00000000005A5A00: Object neither in heap nor stack-allocated in thread load-5
LT  stderr 00000000005A5A00:	O-Slot=0000000000A9E3A8
LT  stderr 00000000005A5A00:	O-Slot value=0000000000141FB0
LT  stderr 00000000005A5A00:	PC=0000FFFF8C19560C
LT  stderr 00000000005A5A00:	framesWalked=1
LT  stderr 00000000005A5A00:	arg0EA=0000000000A9E3C8
LT  stderr 00000000005A5A00:	walkSP=0000000000A9E338
LT  stderr 00000000005A5A00:	literals=0000000000000000
LT  stderr 00000000005A5A00:	jitInfo=0000FFFF8812C7B8
LT  stderr 00000000005A5A00:	method=00000000000616D8 (java/lang/Thread.genThreadName()Ljava/lang/String;) (JIT)
LT  stderr 00000000005A5A00:	stack=0000000000A351C0-0000000000A9EE90
LT  stderr 02:51:56.837 0x148300    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_aarch64_linux_Nightly/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))

pshipton avatar Sep 28 '22 03:09 pshipton

Right, this is the same problem again:

LT  stderr 00000000005A5A00:	O-Slot=0000000000A9E3A8
LT  stderr 00000000005A5A00:	O-Slot value=0000000000141FB0

> !allclasses | grep java/lang/Thread$ThreadNumbering
0x0000000000156100	0x0000000000000328	0x0000FFFFA80B01E8	0x0000FFFF516BA018	0	java/lang/Thread$ThreadNumbering
0x0000FFFF516BA018	0x00000220	0x0000FFFFA80B01E8	0x00000020	0x0C804000	java/lang/Thread$ThreadNumbering
> !j9class 0x0000000000156100 | grep ramStatics
	0x48: UDATA* ramStatics = !j9x 0x0000000000141FB0

dmitripivkine avatar Sep 28 '22 12:09 dmitripivkine

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_0/16 LambdaLoadTest_J9_5m_0 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_0/16/system_test_output.tar.gz

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_2/16 LambdaLoadTest_J9_5m_1 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_2/16/system_test_output.tar.gz

pshipton avatar Sep 29 '22 14:09 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/18/ LambdaLoadTest_CS_5m_0 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/18/system_test_output.tar.gz

pshipton avatar Sep 30 '22 04:09 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_1/18 LambdaLoadTest_J9_5m_1 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_1/18/system_test_output.tar.gz

pshipton avatar Oct 03 '22 19:10 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_1/19 LambdaLoadTest_CS_5m_0 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_linux_Nightly_testList_1/19/system_test_output.tar.gz

pshipton avatar Oct 04 '22 13:10 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/22 LambdaLoadTest_CS_5m_0 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/22/system_test_output.tar.gz

pshipton avatar Oct 06 '22 12:10 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.functional_x86-64_linux_OMR_testList_1/11 testSCCMLTests1_openj9_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.functional_x86-64_linux_OMR_testList_1/11/functional_test_output.tar.gz

Testing: Test 45: LIR 1445.1 : Debug Area PrintStats Test : Ensure debug area sizes are correct in generated javacore
Test start time: 2022/10/07 07:42:00 Atlantic Standard Time
Running command: cat javacore.txt
Time spent starting: 5 milliseconds
00007F91A0131A00: Object neither in heap nor stack-allocated in thread main
00007F91A0131A00:	O-Slot=00007F91A0301F18
00007F91A0131A00:	O-Slot value=00007F91A0426CA8
00007F91A0131A00:	PC=00007F9004EF636D
00007F91A0131A00:	framesWalked=2
00007F91A0131A00:	arg0EA=00007F91A0301F88
00007F91A0131A00:	walkSP=00007F91A0301E00
00007F91A0131A00:	literals=00007F91A024D1B0
00007F91A0131A00:	jitInfo=00007F8FFE94ABB8
00007F91A0131A00:	method=00007F91A023E3F0 (java/lang/Thread.<init>(Ljava/lang/ThreadGroup;Ljava/lang/String;ILjava/lang/Runnable;JLjava/security/AccessControlContext;)V) (JIT)
00007F91A0131A00:	stack=00007F91A02FAE48-00007F91A0302560
10:42:00.014 0x7f8fb0003c00    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_x86-64_linux_OMR/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:109: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))

pshipton avatar Oct 07 '22 16:10 pshipton

Not sure if this is the same problem, but seems to fit.

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/24 TestJlmRemoteClassAuth_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/24/system_test_output.tar.gz

STF 12:38:59.772 - +------ Step 5 - Wait for the processes to complete
STF 12:38:59.772 - | Wait for processes to meet expectations
STF 12:38:59.772 - |   Processes: [LT1, CL1]
STF 12:38:59.772 - |
STF 12:38:59.772 - Monitoring processes: CL1 LT1
CL1 j> 2022/10/08 12:39:00.041 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/10/08 12:39:00.089 Attempting to connect
CL1 j> 2022/10/08 12:39:00.270 Monitored VM not ready at Oct 8, 2022, 12:39:00 PM (attempt 1, elapsed 169ms).
CL1 j> 2022/10/08 12:39:00.270 Waiting 5 secs and trying again...
STF 12:39:02.056 - Found dump at: /Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16651921053862/TestJlmRemoteClassAuth_1/20221008-123858-TestJlmRemoteClassAuth/results/core.20221008.123902.30900.0001.dmp
CL1 stderr core file generated - /Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16651921053862/TestJlmRemoteClassAuth_1/20221008-123858-TestJlmRemoteClassAuth/results/core.20221008.123902.30900.0001.dmp
CL1 j> 2022/10/08 12:39:05.273 Attempting to connect
CL1 j> 2022/10/08 12:39:13.866 Monitored VM not ready at Oct 8, 2022, 12:39:13 PM (attempt 2, elapsed 13777ms).
CL1 j> 2022/10/08 12:39:13.867 Waiting 5 secs and trying again...
STF 12:39:13.990 - **FAILED** Process LT1 has ended unexpectedly

3.LT1.stderr

Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=000000010123AB20 Handler2=00000001013FA210 InaccessibleAddress=9100C29412000028
x0=0000000140973500 x1=0000000140973500 x2=000000016074A600 x3=000000016FE5E920
x4=000000016FE5E8E8 x5=000000016FE5E8D8 x6=000000016FE5E8D0 x7=000000016FE5E8C8
x8=9100C29412000020 x9=00000001200665A8 x10=000000016FE5E8D0 x11=000000016FE5E8D8
x12=000000016FE5E8E8 x13=000000016FE5E8C0 x14=0000000101284094 x15=0000000000000010
x16=0000000101283B30 x17=00000001F5FB9AB0 x18=0000000106AF5B80 x19=0000000040000000
x20=9100C29412000000 x21=000000016FE5E8C0 x22=000000016FE5E8D0 x23=00000001207A32F8
x24=000000016FE5E8E8 x25=000000016FE5E8D8 x26=000000016FE5E8D0 x27=000000016FE5E8C8
x28=000000010135AC64 x29(FP)=000000016FE5E8B0 x30(LR)=0000000101283B24 x31(SP)=000000016FE5E620
PC=0000000101298AAC SP=000000016FE5E620
v0 00000001072ab7d0 (f: 120240080.000000, d: 2.181402e-314)
v1 000000004e800000 (f: 1317011456.000000, d: 6.506901e-315)
v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
v3 3fd0a3250a7390f0 (f: 175345904.000000, d: 2.599576e-01)
v4 0000000000000003 (f: 3.000000, d: 1.482197e-323)
v5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
v6 0000080000000800 (f: 2048.000000, d: 4.345847e-311)
v7 0000000000000004 (f: 4.000000, d: 1.976263e-323)
v8 000000003f6b86d4 (f: 1064011456.000000, d: 5.256915e-315)
v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
v17 3fd556aed1735555 (f: 3513996544.000000, d: 3.334157e-01)
v18 bf3598ab0cff38ca (f: 218052816.000000, d: -3.295343e-04)
v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v21 ffffffffffffffff (f: 4294967296.000000, d: nan)
v22 ffffffffffffffff (f: 4294967296.000000, d: nan)
v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v30 3ff5555555555555 (f: 1431655808.000000, d: 1.333333e+00)
v31 4000000000000000 (f: 0.000000, d: 2.000000e+00)
Module=/Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
Module_base_address=0000000101218000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
Symbol_address=0000000101283B30
Target=2_90_20221008_90 (Mac OS X 11.4)
CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/08 12:39:02 - please wait.
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=000000010123AB20 Handler2=00000001013FA210 InaccessibleAddress=9100C29412000028
x0=000000014095E300 x1=000000014095E300 x2=000000016074A600 x3=000000016FD2E920
x4=000000016FD2E8E8 x5=000000016FD2E8D8 x6=000000016FD2E8D0 x7=000000016FD2E8C8
x8=9100C29412000020 x9=00000001200665A8 x10=000000016FD2E8D0 x11=000000016FD2E8D8
x12=000000016FD2E8E8 x13=000000016FD2E8C0 x14=0000000101284094 x15=0000000000000010
x16=0000000101283B30 x17=000000016001FE81 x18=0000000106AF5B80 x19=0000000040000000
x20=9100C29412000000 x21=000000016FD2E8C0 x22=000000016FD2E8D0 x23=00000001207A32F8
x24=000000016FD2E8E8 x25=000000016FD2E8D8 x26=000000016FD2E8D0 x27=000000016FD2E8C8
x28=000000010135AC64 x29(FP)=000000016FD2E8B0 x30(LR)=0000000101283B24 x31(SP)=000000016FD2E620
PC=0000000101298AAC SP=000000016FD2E620
v0 0000000106b3f188 (f: 112456072.000000, d: 2.177556e-314)
v1 000000004e800000 (f: 1317011456.000000, d: 6.506901e-315)
v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
v3 0000007f0000007f (f: 127.000000, d: 2.694935e-312)
v4 0000000000000003 (f: 3.000000, d: 1.482197e-323)
v5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
v6 0000080000000800 (f: 2048.000000, d: 4.345847e-311)
v7 0000000000000004 (f: 4.000000, d: 1.976263e-323)
v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v16 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v17 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v18 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v21 ffffffffffffffff (f: 4294967296.000000, d: nan)
v22 ffffffffffffffff (f: 4294967296.000000, d: nan)
v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v30 3ff5555555555555 (f: 1431655808.000000, d: 1.333333e+00)
v31 4000000000000000 (f: 0.000000, d: 2.000000e+00)
Module=/Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
Module_base_address=0000000101218000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
Symbol_address=0000000101283B30
Target=2_90_20221008_90 (Mac OS X 11.4)
CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/08 12:39:02 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16651921053862/TestJlmRemoteClassAuth_1/20221008-123858-TestJlmRemoteClassAuth/results/core.20221008.123902.30900.0001.dmp' in response to an event
JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16651921053862/TestJlmRemoteClassAuth_1/20221008-123858-TestJlmRemoteClassAuth/results/core.20221008.123902.30900.0001.dmp
0000000140973500: Object neither in heap nor stack-allocated in thread load-5
0000000140973500:	O-Slot=000000013A81E008
0000000140973500:	O-Slot value=000000012078B224
0000000140973500:	PC=00000001311F476D
0000000140973500:	framesWalked=0
0000000140973500:	arg0EA=000000013A81E0E8
0000000140973500:	walkSP=000000013A81DDC0
0000000140973500:	literals=000000013119ED50
0000000140973500:	jitInfo=0000000000000000
0000000140973500:	method=000000013119ED50 (net/adoptopenjdk/test/lang/invoke/AsTypeTest.testConvertBoolean(Z)V) (Interpreted)
0000000140973500:	stack=000000013A818C48-000000013A81E480

pshipton avatar Oct 11 '22 16:10 pshipton