AArch64 macOS SharedClasses.SCM23.MultiCL_0: Unable to locate JIT stack map

[knn-k]

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/77/

OpenJ9 - 7c5c4148460 OMR - cf8ddbd1adc JCL - d7f7dff2a37 based on jdk-17.0.4+6)

===============================================
Running test SharedClasses.SCM23.MultiCL_0 ...
===============================================
(... lines omitted ...)
STF 11:53:00.145 - Running command: /Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java -Xshareclasses:name=SCC_SysTest_Workload_Cache,cacheDir=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16551701208210/SharedClasses.SCM23.MultiCL_0/20220614-115258-SharedClasses/results/caches -Xaot:forceAoT,count=1 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs -classpath /Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/jvmtest/system/openj9-systemtest/openj9.test.sharedClasses/bin net.openj9.test.sc.LoaderSlaveMultiCL
(... lines omitted ...)
MCL4 11:56:17 >> Loaded 16000 classes...
MCL2 11:56:20 >> Loaded 20000 classes...
MCL2 11:56:20 >> Total classes loaded = 20001
MCL2 stderr JVMCDRT000E Unable to locate JIT stack map - aborting VM
MCL2 stderr JVMCDRT001E Method: java/lang/ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (000000015801F658)
MCL2 stderr JVMCDRT002E Failing PC: 000000014004FFFD (offset 0000000000000001), metaData = 00000001502581E8
STF 11:56:20.549 - **FAILED** Process MCL2 ended with exit code (255) and not the expected exit code/s (0)
STF 11:56:20.550 - Monitoring Report Summary:
STF 11:56:20.550 -   o Process MCL1 is still running
STF 11:56:20.550 -   o Process MCL2 ended with exit code (255) and not the expected exit code/s (0)
STF 11:56:20.550 -   o Process MCL3 is still running
STF 11:56:20.550 -   o Process MCL4 is still running
STF 11:56:20.550 -   o Process MCL5 is still running
STF 11:56:20.550 - Killing processes: MCL1 MCL2 MCL3 MCL4 MCL5

[knn-k]

Another issue with SCM23 on AArch64 macOS: #14621

[knn-k]

Some timeouts in 20x Grinder job: https://openj9-jenkins.osuosl.org/job/Grinder_testList_0/126/

[knn-k]

The timeouts in the Grinder job may be related to #15352.

[pshipton]

Changes in the build where this was seen.

https://github.com/eclipse-openj9/openj9/compare/3d06b2f9c2c...7c5c4148460

[knn-k]

"Unable to locate JIT stack map" was observed in a 20x Grinder run: https://openj9-jenkins.osuosl.org/job/Grinder/1025/ . The job uses the build from June 7 nightly build.

OpenJ9   - 0414be65436
OMR      - c859f0d17d0

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.system_aarch64_mac_Nightly_testList_0/112 SharedClasses.SCM23.MultiCL_0

No diagnostics https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk18_j9_extended.system_aarch64_mac_Nightly_testList_0/112/system_test_output.tar.gz

MCL4 11:50:52 >> Total classes loaded = 20001
MCL4 stderr JVMCDRT000E Unable to locate JIT stack map - aborting VM
MCL4 stderr JVMCDRT001E Method: java/lang/ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (000000012811F788)
MCL4 stderr JVMCDRT002E Failing PC: 00000001389E81D5 (offset 0000000000000001), metaData = 00000001180DF6E8
STF 11:50:53.044 - **FAILED** Process MCL4 ended with exit code (255) and not the expected exit code/s (0)

[knn-k]

"Unable to locate JIT stack map" seems to be thrown as an exception by jitWalkFrame() in com/ibm/j9ddr/vm29/j9/stackwalker/JITStackWalker.java. The method shown in the message is the same: java/lang/ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_mac_Nightly/8 java/lang/String/CompactString/CompareToIgnoreCase.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_aarch64_mac_Nightly/8/openjdk_test_output.tar.gz

21:34:54  STDERR:
21:34:54  STATUS:Passed.
21:34:54  JVMCDRT000E Unable to locate JIT stack map - aborting VM
21:34:54  JVMCDRT001E Method: java/lang/ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (0000000138DB9E40)
21:34:54  JVMCDRT002E Failing PC: 000000016F0463F5 (offset 0000000000000001), metaData = 00000002A8029F68
21:34:54  01:18:48.778 0x15083b900j9codertvm(j9ji.110    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDK19_aarch64_mac_Nightly@2/openj9/runtime/codert_vm/jswalk.c:538: ((0 ))

[knn-k]

The symptom is different, but I guess it is related to the crash in Issue #15518 and some other macOS issues.

• The assertion failure occurred after the test result was printed as "Total tests run: 29, Passes: 29, Failures: 0, Skips: 0". The java process was about to terminate.
• The stack walk shows the method name "java/lang/ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;".

Unfortunately, javacore.20220915.111848.38544.0002.txt for the CompareToIgnoreCase test does not contain meaningful information.

[pshipton]

There is a core file for this one, from the assert. It's not helpful?

[knn-k]

PR #15907 should fix this issue.