openj9
openj9 copied to clipboard
eclipse-openj9
JDK22 JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.ClassFileVersionImpl incompatible with java.lang.classfile.MethodModel
Failure link
From an internal build(cent8le-rtp-rt6-1):
22:03:42 openjdk version "22-internal" 2024-03-19
22:03:42 OpenJDK Runtime Environment (build 22-internal-adhoc.jenkins.BuildJDK22ppc64lelinuxPersonal)
22:03:42 Eclipse OpenJ9 VM (build master-f524f995be4, JRE 22 Linux ppc64le-64-Bit Compressed References 20240215_20 (JIT enabled, AOT enabled)
22:03:42 OpenJ9 - f524f995be4
22:03:42 OMR - 0966eb360f6
22:03:42 JCL - ff0372607f0 based on jdk-22+35)
Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)
23:56:24 ===============================================
23:56:24 Running test CpMpJlink_0 ...
23:56:24 ===============================================
23:56:24 CpMpJlink_0 Start Time: Thu Feb 15 20:56:23 2024 Epoch Time (ms): 1708059383630
23:56:24 variation: Mode150
23:56:24 JVM_OPTIONS: -XX:+UseCompressedOops
23:56:31 JLNK stderr Warning: The 2 argument for --compress is deprecated and may be removed in a future release
23:56:32 JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.ClassFileVersionImpl incompatible with java.lang.classfile.MethodModel
23:56:32 STF 20:56:31.803 - **FAILED** Process JLNK ended with exit code (1) and not the expected exit code/s (0)
23:56:32 STF 20:56:31.803 - Monitoring Report Summary:
23:56:32 STF 20:56:31.803 - o Process JLNK ended with exit code (1) and not the expected exit code/s (0)
23:56:32 STF 20:56:31.803 - Killing processes: JLNK
23:56:32 STF 20:56:31.803 - o Process JLNK pid 1789847 is not running
23:56:32 **FAILED** at step 4 (Run jlink). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk22_j9_sanity.system_ppc64le_linux_Personal_testList_0/aqa-tests/TKG/../TKG/output_17080529032654/CpMpJlink_0/20240215-205623-CpMpJlinkTest/execute.pl line 160.
23:56:32 STF 20:56:31.821 - **FAILED** execute script failed. Expected return value=0 Actual=1
23:56:32 STF 20:56:31.822 -
23:56:32 STF 20:56:31.822 - ==================== T E A R D O W N ====================
23:56:32 STF 20:56:31.822 - Running teardown: perl /home/jenkins/workspace/Test_openjdk22_j9_sanity.system_ppc64le_linux_Personal_testList_0/aqa-tests/TKG/../TKG/output_17080529032654/CpMpJlink_0/20240215-205623-CpMpJlinkTest/tearDown.pl
23:56:32 STF 20:56:31.919 - TEARDOWN stage completed
23:56:32 STF 20:56:31.924 -
23:56:32 STF 20:56:31.925 - ===================== R E S U L T S =====================
23:56:32 STF 20:56:31.925 - Stage results:
23:56:32 STF 20:56:31.925 - setUp: pass
23:56:32 STF 20:56:31.925 - execute: *fail*
23:56:32 STF 20:56:31.925 - teardown: pass
23:56:32 STF 20:56:31.925 -
23:56:32 STF 20:56:31.925 - Overall result: **FAILED**
23:56:32 -----------------------------------
23:56:32 CpMpJlink_0_FAILED
50x internal grinder - https://github.com/eclipse-openj9/openj9/issues/18972#issuecomment-1962035006
4 machines passed, failed 1/10 on cent8le-svl-rt2-1
16:50:31 JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.AccessFlagsImpl incompatible with java.lang.classfile.MethodModel
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.system_ppc64le_linux_Nightly_testList_0/163
CpMpJlink_1 -XX:-UseCompressedOops
JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.BoundAttribute$BoundBootstrapMethodsAttribute incompatible with jdk.internal.classfile.ClassElement
https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.system_ppc64_aix_Release_testList_2/1
CpMpJlink_1
JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.BoundAttribute$BoundInnerClassesAttribute incompatible with java.lang.classfile.MethodModel
I am unable to reproduce the issue locally on a ppc64le linux machine cent8le-rtp-rt6-1, which is where the issue occurred in the past, after running the test CpMpJlinkTest with aqa-systemtest for 1000 iterations.
The intermittent CpMpJlinkTest failure is reproducible in a grinder that runs the test for 1000 iterations on a ppc64le linux machine. The issue also occurs in a grinder with -Xint.
I wonder if the -Xint parameter is getting to the JVM where the ClassCastException occurs.
It seems like -Xint is not getting to the jvm that is run by the jlink command in CpMpJlinkTest, which is where the ClassCastException occurs, when creating an image. jlink does not seem to have an option for specifying -Xint to the jvm it runs when creating an image. The java command in CpMpJlinkTest, which runs the image created by jlink, gets the -Xint that is specified by EXTRA_OPTIONS in the grinder configuration.
The OPENJ9_JAVA_OPTIONS environment variable was set to -Xint in a modified CpMpJlinkTest to ensure -Xint gets to the jvm that is run by the jlink command in the test. However, the intermittent issue occurs in a grinder that runs the test for 1000 iterations on a ppc64le linux machine with the modified test and with -Xint in the EXTRA_OPTIONS of the grinder configuration.
The jvm that is run by the jlink command in CpMpJlinkTest gets -Xint when it is enabled by default in the jvm. However, the jlink command in the test does not get -Xint when it is specified by the OPENJ9_JAVA_OPTIONS environment variable in the test or when it is specified by EXTRA_OPTIONS in the grinder configuration.
The results of a grinder that runs CpMpJlinkTest 1000 times on a ppc64le linux machine with a jdk that enables -Xint by default show no intermittent ClassCastException errors. Additionally, the results indicate that the jlink command in the test takes approximately 50 seconds to complete instead of the expected 10 seconds.
I was able to generate a core file for ClassCastException: jdk.internal.classfile.impl.BoundAttribute$BoundAnnotationDefaultAttr incompatible with java.lang.classfile.CodeModel in a grinder which utilized a jdk that enables -Xdump by default. The core file indicates a potential jit issue, where the top frames in the stackslots of the main thread are jit frames.
> !threads
!stack 0x00095800 !j9vmthread 0x00095800 !j9thread 0x7fff9801ce60 tid 0x23c9ac (2345388) // (main)
!stack 0x00097d00 !j9vmthread 0x00097d00 !j9thread 0x7fff980f6ef0 tid 0x23c9ae (2345390) // (JIT Compilation Thread-000 Suspended)
!stack 0x0009a200 !j9vmthread 0x0009a200 !j9thread 0x7fff980f7468 tid 0x23c9af (2345391) // (JIT Compilation Thread-001)
!stack 0x0009c800 !j9vmthread 0x0009c800 !j9thread 0x7fff980f8420 tid 0x23c9b0 (2345392) // (JIT Compilation Thread-002 Suspended)
!stack 0x0009ed00 !j9vmthread 0x0009ed00 !j9thread 0x7fff980f8998 tid 0x23c9b1 (2345393) // (JIT Compilation Thread-003 Suspended)
!stack 0x000a1300 !j9vmthread 0x000a1300 !j9thread 0x7fff980f9950 tid 0x23c9b2 (2345394) // (JIT Compilation Thread-004 Suspended)
!stack 0x000a3800 !j9vmthread 0x000a3800 !j9thread 0x7fff980f9ec8 tid 0x23c9b3 (2345395) // (JIT Compilation Thread-005 Suspended)
!stack 0x000a5e00 !j9vmthread 0x000a5e00 !j9thread 0x7fff980fae80 tid 0x23c9b4 (2345396) // (JIT Compilation Thread-006 Suspended)
!stack 0x000a8300 !j9vmthread 0x000a8300 !j9thread 0x7fff980fb3f8 tid 0x23c9b5 (2345397) // (JIT Diagnostic Compilation Thread-007 Suspended)
!stack 0x000aa800 !j9vmthread 0x000aa800 !j9thread 0x7fff98161250 tid 0x23c9b6 (2345398) // (JIT-SamplerThread)
!stack 0x000ace00 !j9vmthread 0x000ace00 !j9thread 0x7fff981617c8 tid 0x23c9b7 (2345399) // (IProfiler)
!stack 0x001bd600 !j9vmthread 0x001bd600 !j9thread 0x7fff981acbe0 tid 0x23c9b8 (2345400) // (Common-Cleaner)
!stack 0x0023f800 !j9vmthread 0x0023f800 !j9thread 0x7fff20000cc0 tid 0x23c9ba (2345402) // (Finalizer thread)
!stack 0x00241d00 !j9vmthread 0x00241d00 !j9thread 0x7fff20001238 tid 0x23c9bb (2345403) // (Concurrent Mark Helper)
!stack 0x00244300 !j9vmthread 0x00244300 !j9thread 0x7fff9827da60 tid 0x23c9bc (2345404) // (GC Worker)
!stack 0x00246800 !j9vmthread 0x00246800 !j9thread 0x7fff9827dfd8 tid 0x23c9bd (2345405) // (GC Worker)
!stack 0x00248d00 !j9vmthread 0x00248d00 !j9thread 0x7fff9827ed10 tid 0x23c9be (2345406) // (GC Worker)
!stack 0x00285b00 !j9vmthread 0x00285b00 !j9thread 0x7fff00016a30 tid 0x23c9c0 (2345408) // (Attach API update file access time)
!stack 0x00288000 !j9vmthread 0x00288000 !j9thread 0x7fff00016fa8 tid 0x23c9c1 (2345409) // (Attach API wait loop)
> !stack 0x00095800
<95800> known but unhandled frame type com.ibm.j9ddr.vm29.pointer.U8Pointer @ 0x00000005
FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT
<95800> !j9method 0x0000000000416F10 jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform.lambda$resolve$0(Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V
<95800> !j9method 0x0000000000420888 jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/Object;)V
<95800> !j9method 0x0000000000420F20 jdk/internal/classfile/impl/ChainedMethodBuilder.with(Ljava/lang/classfile/MethodElement;)Ljava/lang/classfile/MethodBuilder;
<95800> !j9method 0x0000000000420FC0 jdk/internal/classfile/impl/ChainedMethodBuilder.with(Ljava/lang/classfile/ClassFileElement;)Ljava/lang/classfile/ClassFileBuilder;
<95800> !j9method 0x00000000004153F0 java/lang/classfile/MethodTransform.lambda$dropping$0(Ljava/util/function/Predicate;Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V
<95800> !j9method 0x0000000000415A50 java/lang/classfile/MethodTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/classfile/ClassFileBuilder;Ljava/lang/classfile/ClassFileElement;)V
<95800> !j9method 0x00000000004153D0 java/lang/classfile/MethodTransform.lambda$resolve$1(Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V
<95800> !j9method 0x0000000000421068 java/lang/classfile/MethodTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/Object;)V
<95800> !j9method 0x000000000040FAE0 jdk/internal/classfile/impl/MethodImpl.forEachElement(Ljava/util/function/Consumer;)V
<95800> !j9method 0x00000000003CB878 java/lang/classfile/ClassFileBuilder.transform(Ljava/lang/classfile/CompoundElement;Ljava/lang/classfile/ClassFileTransform;)V
<95800> !j9method 0x00000000003D7AA0 jdk/internal/classfile/impl/DirectClassBuilder.transformMethod(Ljava/lang/classfile/MethodModel;Ljava/lang/classfile/MethodTransform;)Ljava/lang/classfile/ClassBuilder;
<95800> !j9method 0x0000000000418620 jdk/internal/classfile/impl/TransformImpl$ClassMethodTransform.lambda$resolve$0(Ljava/lang/classfile/ClassBuilder;Ljava/lang/classfile/ClassElement;)V
<95800> !j9method 0x000000000041B6F0 jdk/internal/classfile/impl/TransformImpl$ClassMethodTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/Object;)V
<95800> !j9method 0x000000000041C708 jdk/internal/classfile/impl/ChainedClassBuilder.with(Ljava/lang/classfile/ClassElement;)Ljava/lang/classfile/ClassBuilder;
<95800> !j9method 0x000000000041C7E8 jdk/internal/classfile/impl/ChainedClassBuilder.with(Ljava/lang/classfile/ClassFileElement;)Ljava/lang/classfile/ClassFileBuilder;
<95800> !j9method 0x0000000000410358 java/lang/classfile/ClassTransform.lambda$dropping$0(Ljava/util/function/Predicate;Ljava/lang/classfile/ClassBuilder;Ljava/lang/classfile/ClassElement;)V
<95800> !j9method 0x00000000004088C8 java/lang/classfile/ClassTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/classfile/ClassFileBuilder;Ljava/lang/classfile/ClassFileElement;)V
<95800> !j9method 0x00000000004102F8 java/lang/classfile/ClassTransform.lambda$resolve$3(Ljava/lang/classfile/ClassBuilder;Ljava/lang/classfile/ClassElement;)V
<95800> !j9method 0x000000000041CAE0 java/lang/classfile/ClassTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/Object;)V
<95800> !j9method 0x00000000000C5BD0 java/lang/Iterable.forEach(Ljava/util/function/Consumer;)V
<95800> !j9method 0x000000000040BCD8 jdk/internal/classfile/impl/ClassImpl.forEachElement(Ljava/util/function/Consumer;)V
<95800> !j9method 0x00000000003CB878 java/lang/classfile/ClassFileBuilder.transform(Ljava/lang/classfile/CompoundElement;Ljava/lang/classfile/ClassFileTransform;)V
<95800> !j9method 0x000000000041A930 jdk/internal/classfile/impl/ClassFileImpl$2.accept(Ljava/lang/classfile/ClassBuilder;)V
<95800> !j9method 0x000000000041A950 jdk/internal/classfile/impl/ClassFileImpl$2.accept(Ljava/lang/Object;)V
<95800> !j9method 0x00000000003C4210 jdk/internal/classfile/impl/ClassFileImpl.build(Ljava/lang/classfile/constantpool/ClassEntry;Ljava/lang/classfile/constantpool/ConstantPoolBuilder;Ljava/util/function/Consumer;)[B
<95800> !j9method 0x00000000003C4230 jdk/internal/classfile/impl/ClassFileImpl.transform(Ljava/lang/classfile/ClassModel;Ljava/lang/classfile/constantpool/ClassEntry;Ljava/lang/classfile/ClassTransform;)[B
<95800> !j9method 0x00000000003C3178 java/lang/classfile/ClassFile.transform(Ljava/lang/classfile/ClassModel;Ljava/lang/classfile/ClassTransform;)[B
<95800> !j9method 0x00000000002A7C98 jdk/tools/jlink/internal/plugins/StripJavaDebugAttributesPlugin.lambda$transform$3(Ljdk/tools/jlink/plugin/ResourcePoolEntry;)Ljdk/tools/jlink/plugin/ResourcePoolEntry;
<95800> !j9method 0x00000000003B7D48 jdk/tools/jlink/internal/plugins/StripJavaDebugAttributesPlugin$$Lambda/0x00000000983355f8.apply(Ljava/lang/Object;)Ljava/lang/Object;
<95800> !j9method 0x0000000000391AC0 jdk/tools/jlink/plugin/ResourcePool.lambda$transformAndCopy$0(Ljava/util/function/Function;Ljdk/tools/jlink/plugin/ResourcePoolBuilder;Ljdk/tools/jlink/plugin/ResourcePoolEntry;)V
<95800> !j9method 0x00000000003B6588 jdk/tools/jlink/plugin/ResourcePool$$Lambda/0x0000000098334540.accept(Ljava/lang/Object;)V
<95800> !j9method 0x0000000000115940 java/util/Iterator.forEachRemaining(Ljava/util/function/Consumer;)V
<95800> !j9method 0x00000000001C4B50 java/util/Spliterators$IteratorSpliterator.forEachRemaining(Ljava/util/function/Consumer;)V
<95800> !j9method 0x00000000001C94F0 java/util/stream/ReferencePipeline$Head.forEach(Ljava/util/function/Consumer;)V
<95800> !j9method 0x0000000000391AA0 jdk/tools/jlink/plugin/ResourcePool.transformAndCopy(Ljava/util/function/Function;Ljdk/tools/jlink/plugin/ResourcePoolBuilder;)V
<95800> !j9method 0x00000000002A7C78 jdk/tools/jlink/internal/plugins/StripJavaDebugAttributesPlugin.transform(Ljdk/tools/jlink/plugin/ResourcePool;Ljdk/tools/jlink/plugin/ResourcePoolBuilder;)Ljdk/tools/jlink/plugin/ResourcePool;
<95800> !j9method 0x00000000002A69E8 jdk/tools/jlink/internal/plugins/DefaultStripDebugPlugin.transform(Ljdk/tools/jlink/plugin/ResourcePool;Ljdk/tools/jlink/plugin/ResourcePoolBuilder;)Ljdk/tools/jlink/plugin/ResourcePool;
<95800> !j9method 0x000000000038DDB8 jdk/tools/jlink/internal/ImagePluginStack.visitResources(Ljdk/tools/jlink/internal/ResourcePoolManager;)Ljdk/tools/jlink/plugin/ResourcePool;
<95800> !j9method 0x000000000038F478 jdk/tools/jlink/internal/ImageFileCreator.generateJImage(Ljdk/tools/jlink/internal/ResourcePoolManager;Ljdk/tools/jlink/internal/BasicImageWriter;Ljdk/tools/jlink/internal/ImagePluginStack;Ljava/io/DataOutputStream;)Ljdk/tools/jlink/plugin/ResourcePool;
<95800> !j9method 0x000000000038F458 jdk/tools/jlink/internal/ImageFileCreator.writeImage(Ljava/util/Set;Ljava/nio/ByteOrder;)V
<95800> !j9method 0x000000000038F3F8 jdk/tools/jlink/internal/ImageFileCreator.create(Ljava/util/Set;Ljava/nio/ByteOrder;Ljdk/tools/jlink/internal/ImagePluginStack;)Ljdk/tools/jlink/internal/ExecutableImage;
<95800> !j9method 0x0000000000306238 jdk/tools/jlink/internal/JlinkTask$ImageHelper.retrieve(Ljdk/tools/jlink/internal/ImagePluginStack;)Ljdk/tools/jlink/internal/ExecutableImage;
<95800> !j9method 0x000000000038DD58 jdk/tools/jlink/internal/ImagePluginStack.operate(Ljdk/tools/jlink/internal/ImagePluginStack$ImageProvider;)V
<95800> !j9method 0x000000000027EA48 jdk/tools/jlink/internal/JlinkTask.createImage(Ljdk/tools/jlink/internal/Jlink$JlinkConfiguration;)V
<95800> !j9method 0x000000000027E9C8 jdk/tools/jlink/internal/JlinkTask.run([Ljava/lang/String;)I
<95800> !j9method 0x000000000027C558 jdk/tools/jlink/internal/Main.run(Ljava/io/PrintWriter;Ljava/io/PrintWriter;[Ljava/lang/String;)I
<95800> !j9method 0x000000000027C538 jdk/tools/jlink/internal/Main.main([Ljava/lang/String;)V
<95800> JNI call-in frame
<95800> Native method frame
> !stackslots 0x00095800
<95800> *** BEGIN STACK WALK, flags = 00400001 walkThread = 612,352 ***
<95800> ITERATE_O_SLOTS
<95800> RECORD_BYTECODE_PC_OFFSET
<95800> Initial values: walkSP = 0x0000000000192EC8, PC = 0x0000000000000005, literals = 0x0000000000000000, A0 = 0x0000000000192EE8, j2iFrame = 0x0000000000193350, decomp = 0x0000000000000000
<95800> JIT resolve frame: bp = 0x0000000000192EE8, sp = 0x0000000000192EC8, pc = 0x0000000000000005, cp = 0x0000000000000000, arg0EA = 0x0000000000192EE8, flags = 0x0000000000880000
<95800> Object push (savedJITException)
<95800> O-Slot[0x0000000000192EC8] = 0x0000000000000000
<95800> Runtime helper resolve
<95800> unwindSP initialized to 0x0000000000192EF0
<95800> JIT-Resolve-RegisterMap[0x00007FFF9C28D850] = UDATA(0x0000000088D35B00) (jit_r0)
...
<95800> JIT frame: bp = 0x0000000000192F28, pc = 0x00007FFF792F6D4C, unwindSP = 0x0000000000192EF0, cp = 0x0000000000416A10, arg0EA = 0x0000000000192F40, jitInfo = 0x00007FFF70A2EB38
<95800> Method: jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform.lambda$resolve$0(Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V !j9method 0x0000000000416F10
<95800> Bytecode index = 8, inlineDepth = 0, PC offset = 0x000000000000027C
<95800> stackMap=0x00007FFF70A2EC91, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFF8)
<95800> Described JIT args starting at 0x0000000000192F30 for U16(0x0003) slots
<95800> I-Slot: : a2[0x0000000000192F30] = 0x000000000040D700
<95800> I-Slot: : a1[0x0000000000192F38] = 0x00007FFF78ED00EC
<95800> O-Slot: : a0[0x0000000000192F40] = 0x00000000FFF64680
<95800> Described JIT temps starting at 0x0000000000192F20 for IDATA(0x0000000000000001) slots
<95800> O-Slot: : t0[0x0000000000192F20] = 0x0000000000000000
<95800> JIT-RegisterMap = UDATA(0x0000000000000003)
<95800> JIT-RegisterMap-O-Slot[0x00007FFF9C28D948] = 0x00000000FFF64E70 (jit_r31)
...
<95800> JIT frame: bp = 0x0000000000192F68, pc = 0x00007FFF792E3B04, unwindSP = 0x0000000000192F30, cp = 0x00000000004207D0, arg0EA = 0x0000000000192F78, jitInfo = 0x00007FFF70A200F8
<95800> Method: jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform$$Lambda/0x0000000000000000.accept(Ljava/lang/Object;)V !j9method 0x0000000000420888
<95800> Bytecode index = 12, inlineDepth = 0, PC offset = 0x000000000000008C
<95800> stackMap=0x00007FFF70A20200, slots=I16(0x0002) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0002), localBaseOffset=I16(0x0000)
<95800> Described JIT args starting at 0x0000000000192F70 for U16(0x0002) slots
<95800> O-Slot: : a1[0x0000000000192F70] = 0x00000000FFF64E70
<95800> O-Slot: : a0[0x0000000000192F78] = 0x00000000FFF64C58
<95800> Described JIT temps starting at 0x0000000000192F68 for IDATA(0x0000000000000000) slots
<95800> JIT-RegisterMap = UDATA(0x0000000000000000)
<95800> JIT-Frame-RegisterMap[0x00007FFF9C28D8D0] = UDATA(0x00007FFF70970038) (jit_r16)
...
@hzongaro I would like your input for addressing this issue. The core files that were generated in a grinder due to a ClassCastException indicates a potential jit issue, where the top frames in the stackslots of the main thread, which is where the exception was thrown, are related to the jit.
I'm having some difficulty matching up the contents of one of the jitdump files that was produced with one of the core files, so I'm trying another internal grinder run with a version of aqa-tests that adds a TR_Options. I'm hoping to get a trace log for the actual compilation of TransformImpl$MethodCodeTransform.lambda$resolve$0 in which the ClassCastException occurs.
I was able to reproduce this running locally on a ppc64le Linux system. It might be AOT-related. I'll spend some time trying to narrow things down.
Any new news on this one?
I haven't been able to spend further time on this one recently.
@hzongaro we won't hold the release for this, pls move it out of the Java 22 milestone.
@hzongaro how can I know the status of the issue?
Sorry - I missed your last message, @singh264. I haven't had a chance to investigate this one further. I'll pass this along to another developer on the JIT team to investigate further. Whoever is assigned to investigate will post updates as they become available.
@a7ehuo, may I ask you to look at this failure? I will share in Box a build and tests that I had used to reproduce the problem.
- Ran
CpMpJlink_0internally in grinder41536onppc64le_linuxwith latest OpenJ9/OMR360xall passed
- Also locally
100xall passed
I will take a look at the logs @hzongaro collected previously and see if I find anything
Grinder run:
openjdk version "22.0.1-internal" 2024-04-16
OpenJDK Runtime Environment (build 22.0.1-internal-adhoc.jenkins.BuildJDK22ppc64lelinuxPersonal)
Eclipse OpenJ9 VM (build master-69cba86c0f1, JRE 22 Linux ppc64le-64-Bit Compressed References 20240624_58 (JIT enabled, AOT enabled)
OpenJ9 - 69cba86c0f1
OMR - 770936378ec
JCL - ac3fb0c01e2 based on jdk-22.0.1+8)
Local run:
openjdk version "22.0.1-internal" 2024-04-16
OpenJDK Runtime Environment (build 22.0.1-internal-adhoc.root.openj9-openjdk-jdk22)
Eclipse OpenJ9 VM (build master-6036c034f, JRE 22 Linux ppc64le-64-Bit Compressed References 20240624_000000 (JIT enabled, AOT enabled)
OpenJ9 - 6036c034f
OMR - c2719521f
JCL - ac3fb0c01e2 based on jdk-22.0.1+8)
I looked at the logs from CpMpJlink_0_ITER_213. The checkcast error is
JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.BoundAttribute$BoundAnnotationDefaultAttr incompatible with java.lang.classfile.CodeModel.
There are two jit dmp logs from jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform.lambda$resolve$0(Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V, but they don't match the two cores generated [1]. The IL trees from both jitdmp look the same. This method is compiled at cold.
n5n BBStart <block_2> (freq 9973) [ 0x7fff71ff4950] bci=[-1,0,202] rc=0 vc=182 vn=- li=2 udi=- nc=1
n227n GlRegDeps [ 0x7fff720936b0] bci=[-1,0,202] rc=1 vc=182 vn=- li=- udi=- nc=2
n28n aRegLoad gr4 builder<parm 1 Ljava/lang/classfile/MethodBuilder;>[#404 Parm] [flags 0xc0000107 0x0 ] (X>=0 ) [ 0x7fff71ff5080] bci=[-1,27,206] rc=3 vc=182 vn=- li=4 udi=- nc=0 flg=0x100
n7n aRegLoad gr5 me<parm 2 Ljava/lang/classfile/MethodElement;>[#405 Parm] [flags 0xc0000107 0x0 ] [ 0x7fff71ff49f0] bci=[-1,0,202] rc=4 vc=182 vn=- li=2 udi=- nc=0
n13n ificmpne --> block_3 BBStart at n11n () [ 0x7fff71ff4bd0] bci=[-1,4,202] rc=0 vc=182 vn=- li=2 udi=- nc=2 flg=0x20
n9n instanceof jitInstanceOf[#87 helper Method] [flags 0x400 0x0 ] [ 0x7fff71ff4a90] bci=[-1,1,202] rc=1 vc=182 vn=- li=2 udi=- nc=2
n7n ==>aRegLoad
n8n loadaddr java/lang/classfile/CodeModel[#406 Static] [flags 0x18307 0x0 ] [ 0x7fff71ff4a40] bci=[-1,1,202] rc=1 vc=182 vn=- li=2 udi=- nc=0
n10n iconst 0 (X==0 X>=0 X<=0 ) [ 0x7fff71ff4ae0] bci=[-1,4,202] rc=2 vc=182 vn=- li=2 udi=- nc=0 flg=0x302
...
...
n178n BBStart <block_22> (freq 10000) [ 0x7fff72092760] bci=[-1,35,208] rc=0 vc=182 vn=- li=22 udi=- nc=0
n33n return [ 0x7fff71ff5210] bci=[-1,35,208] rc=0 vc=182 vn=- li=22 udi=- nc=0
n4n BBEnd </block_22> ===== [ 0x7fff71ff4900] bci=[-1,35,208] rc=0 vc=182 vn=- li=22 udi=- nc=0
n11n BBStart <block_3> (freq 3830) [ 0x7fff71ff4b30] bci=[-1,7,202] rc=0 vc=182 vn=- li=3 udi=- nc=0
n16n checkcast [#86] [ 0x7fff71ff4cc0] bci=[-1,8,202] rc=0 vc=182 vn=- li=3 udi=- nc=2
n14n aload me<parm 2 Ljava/lang/classfile/MethodElement;>[#405 Parm] [flags 0xc0000107 0x0 ] [ 0x7fff71ff4c20] bci=[-1,7,202] rc=3 vc=182 vn=- li=3 udi=- nc=0
n15n loadaddr java/lang/classfile/CodeModel[#406 Static] [flags 0x18307 0x0 ] [ 0x7fff71ff4c70] bci=[-1,8,202] rc=1 vc=182 vn=- li=3 udi=- nc=0
...
There is only one checkcast (n16n) against java/lang/classfile/CodeModel in block_3 in pre instruction selection tree. One thing is that it first has to pass the instanceof node (n9n) against java/lang/classfile/CodeModel in block_2 to get to the checkcast (n16n) in block_3.
One thing is that the checkcast loads the object me from the memory instead of from the register gr5 which is used by the instanceof node. I see the object loaded is in a collected reference register &GPR_, which looks good to me. So far I haven't found anything incorrect from the collect logs.
n14n ( 2) aload me<parm 2 Ljava/lang/classfile/MethodElement;>[#405 Parm] [flags 0xc0000107 0x0 ] (in &GPR_ 0x7fff721578c0) [ 0x7fff71ff4c20] bci=[-1,7,202] rc=2 vc=182 vn=- li=3 udi=30912 nc=0
[1] core.20240322.115524.2345387.0001.dmp
!stackslots 0x00095800
<95800> JIT frame: bp = 0x0000000000192F28, pc = 0x00007FFF792F6D4C, unwindSP = 0x0000000000192EF0, cp = 0x0000000000416A10, arg0EA = 0x0000000000192F40, jitInfo = 0x00007FFF70A2EB38
<95800> Method: jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform.lambda$resolve$0(Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V !j9method 0x0000000000416F10
<95800> Bytecode index = 8, inlineDepth = 0, PC offset = 0x000000000000027C
<95800> stackMap=0x00007FFF70A2EC91, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFF8)
core.20240322.115527.2345387.0004.dmp
<95800> JIT frame: bp = 0x0000000000192F28, pc = 0x00007FFF792F6D4C, unwindSP = 0x0000000000192EF0, cp = 0x0000000000416A10, arg0EA = 0x0000000000192F40, jitInfo = 0x00007FFF70A2EB38
<95800> Method: jdk/internal/classfile/impl/TransformImpl$MethodCodeTransform.lambda$resolve$0(Ljava/lang/classfile/MethodBuilder;Ljava/lang/classfile/MethodElement;)V !j9method 0x0000000000416F10
<95800> Bytecode index = 8, inlineDepth = 0, PC offset = 0x0000000000013FD8
<95800> stackMap=0x00007FFF70A2EC91, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFF8)
@a7ehuo how can I request a grinder that runs CpMpJlinkTest for 1000 iterations on a ppc64le linux machine to confirm the analysis of the collected logs?
It seems like the next step is to start a grinder that runs CpMpJlinkTest for 1000 iterations on a ppc64le linux machine to confirm the analysis of the collected logs.
@a7ehuo how can I request a grinder that runs CpMpJlinkTest for 1000 iterations on a ppc64le linux machine to confirm the analysis of the collected logs?
@singh264 No need to run it in Grinders. I'll post an update next
First I'm not able to reproduce this issue after disabling AOT or on X86. The issue looks specific to AOT compilation on Power.
One reproduced case on Power looks like below:
JLNK Error: java.lang.ClassCastException: jdk.internal.classfile.impl.FieldImpl incompatible with java.lang.classfile.MethodModel
Based on the CFG, checkcast doesn’t get to run unless it passes instanceof (n13n) check on the same cast class first. The instanceof sequence [1] checks: (1) If the object is NULL (2) Compares object class against guess class (3) check object class castClassCache with the cast class (4) call helper jitInstanceOf
In step (2) of the instanceof sequence, there is a relocation record generated for instruction offset 009c (or Offset(M)=00ec) which is the first instruction of loadAddressConstant of guessClassArray[0] in genInlineTest in non-SVM AOT compilation.
TR_FixedSequenceAddress2 (10) /root/home/ahuo/src/openj9-openjdk-jdk22/omr/compiler/p/codegen/OMRCodeGenerator.cpp 2064 00ec 009c 0000766DECFF4B60
TargetAddress1:0000000000424000, TargetAddress2:0000000000000001
@dsouzai helped look at the logs and found the relocation record for this TR_FixedSequenceAddress2 is incorrect.
TR_RelocationRecord 0000766E4E523B08
size 12 type 10 flags 0 reloFlags 1
offset 82371100
preparePrivateData: addressToPatch: 0000000000422F00
reloLocation: from 0000766E4E523B18 at 0000766E7E0B1EEC (offset ec)
The address to patch becomes 0x422F00 which is jdk/internal/classfile/impl/FieldImpl instead of 0x424000 jdk/internal/classfile/impl/MethodImpl. This explains why in runtime, it passes the instanceof check with an instance of jdk.internal.classfile.impl.FieldImpl against jdk/internal/classfile/impl/MethodImpl in step (2) of the instanceof sequence, but then it fails checkcast next.
Irwin pointed out in non-SVM AOT, the code generated in genInlineTest that checks object class against guess class should not be generated on Power. TR_FixedSequenceAddress2 relocation record is an offset from the startPC, which shouldn't be used for class address.
@zl-wang Could the Power team take a further look at how Power CG genInlineTest should be updated in the non-SVM AOT in this case? Thank you very much!
[1]
------------------------------
n13n ( 0) ificmpne --> block_3 BBStart at n11n () [ 0x766decff4ca0] bci=[-1,4,-] rc=0 vc=294 vn=- li=2 udi=- nc=3 flg=0x20
n9n ( 0) instanceof jitInstanceOf[#87 helper Method] [flags 0x400 0x0 ] (in GPR_0039) [ 0x766decff4b60] bci=[-1,1,-] rc=0 vc=294 vn=- li=2 udi=30912 nc=2
n80n ( 1) ==>aRegLoad (in &GPR_0032) (SeenRealReference )
n8n ( 0) ==>loadaddr (in GPR_0036)
n10n ( 0) iconst 0 (X==0 X>=0 X<=0 ) [ 0x766decff4bb0] bci=[-1,4,-] rc=0 vc=294 vn=- li=2 udi=- nc=0 flg=0x302
n82n ( 0) GlRegDeps [ 0x766ded070960] bci=[-1,4,-] rc=0 vc=294 vn=- li=2 udi=- nc=2
n80n ( 1) ==>aRegLoad (in &GPR_0032) (SeenRealReference )
n81n ( 1) ==>aRegLoad (in &GPR_0033)
------------------------------
[ 0x766ded0e7820] 1 ori GPR_0038, gr15, 0x0
[ 0x766ded0e79e0] 1 ori &GPR_0040, &GPR_0032, 0x0
[ 0x766ded0e81b0] 1 ld D_GPR_0042, [gr16, 688]
[ 0x766ded0e8460] 1 assocreg
PRE:
POST: [GPR_0034 : gr11]
[ 0x766ded0e86e0] 0 cmpldi CCR_0049, &GPR_0040, 0 (1)
[ 0x766ded0e8810] 1 beq CCR_0049, Label L0049
[ 0x766ded0e8990] 1 lwz D_GPR_0045, [&GPR_0040, 0]
[ 0x766ded0e8a80] 1 rlwinm D_GPR_0045, D_GPR_0045, 0000000000000000, FFFFFFFFFFFFFF00
[ 0x766ded0e8c50] 1 lis D_GPR_0044, 0000000000000000
[ 0x766ded0e8cf0] 1 ori D_GPR_0044, D_GPR_0044, 0
[ 0x766ded0e8d90] 1 rldicr D_GPR_0044, D_GPR_0044, 0000000000000020, FFFFFFFF00000000
[ 0x766ded0e8e40] 1 oris D_GPR_0044, D_GPR_0044, 0
[ 0x766ded0e8ee0] 1 ori D_GPR_0044, D_GPR_0044, 0
[ 0x766ded0e9010] 1 cmpld CCR_0049, D_GPR_0045, D_GPR_0044 (2)
[ 0x766ded0e90e0] 1 bne CCR_0049, Snippet Label L0064 (3)
[ 0x766ded0e91d0] 1 b Label L0017
[ 0x766ded0e84f0] 1 Label L0050:
[ 0x766ded0e8250] 1 bl 0000766E9AFAD120 ; Direct Call "jitInstanceOf" (4)
0x766e7e0b2fd0 00000080 [ 0x766ded0e7820] 61e30000 1 ori gr3, gr15, 0x0
0x766e7e0b2fd4 00000084 [ 0x766ded0e79e0] 63e50000 1 ori gr5, gr31, 0x0
0x766e7e0b2fd8 00000088 [ 0x766ded0e81b0] e99002b0 1 ld gr12, [gr16, 688]
0x766e7e0b2fdc 0000008c [ 0x766ded0e8460] 1 assocreg
PRE:
POST: [GPR_0034 : gr11]
0x766e7e0b2fdc 0000008c [ 0x766ded0e86e0] 283f0000 0 cmpldi cr0, gr31, 0
0x766e7e0b2fe0 00000090 [ 0x766ded0e8810] 41820038 1 beq cr0, Label L0049
0x766e7e0b2fe4 00000094 [ 0x766ded0e8990] 80ff0000 1 lwz gr7, [gr31, 0]
0x766e7e0b2fe8 00000098 [ 0x766ded0e8a80] 54e7002e 1 rlwinm gr7, gr7, 0000000000000000, FFFFFFFFFFFFFF00
0x766e7e0b2fec 0000009c [ 0x766ded0e8c50] 3cc00000 1 lis gr6, 0000000000000000
0x766e7e0b2ff0 000000a0 [ 0x766ded0e8cf0] 60c60000 1 ori gr6, gr6, 0
0x766e7e0b2ff4 000000a4 [ 0x766ded0e8d90] 78c607c6 1 rldicr gr6, gr6, 0000000000000020, FFFFFFFF00000000
0x766e7e0b2ff8 000000a8 [ 0x766ded0e8e40] 64c60000 1 oris gr6, gr6, 0
0x766e7e0b2ffc 000000ac [ 0x766ded0e8ee0] 60c60000 1 ori gr6, gr6, 0
0x766e7e0b3000 000000b0 [ 0x766ded0e9010] 7c273040 1 cmpld cr0, gr7, gr6
0x766e7e0b3004 000000b4 [ 0x766ded0e90e0] 4082023c 1 bne cr0, Snippet Label L0064
0x766e7e0b3008 000000b8 [ 0x766ded0e91d0] 48000078 1 b Label L0017
0x766e7e0b300c 000000bc [ 0x766ded0e84f0] 1 Label L0050:
0x766e7e0b300c 000000bc [ 0x766ded0e8250] 481abcd5 1 bl 0000766E7E25ECE0 ; Direct Call "jitInstanceOf"
isn't SVM the default? I guessed it is not for this instance of running @rmnattas it looks like we have to disable using the guessClassArray altogether (the previous loop and the line @a7ehuo pointed out), since they cannot be relocated properly in non-SVM mode.
