cmdLineTester_jvmtitests_hcr_OSRG_nongold failures with JITServer on x86_64 and JDK11

[cjjdespres]

The tests cmdLineTester_jvmtitests_hcr_OSRG_nongold_0 and cmdLineTester_jvmtitests_hcr_OSRG_nongold_2 have started failing in the JITServer nightly tests. Both fail in the rtc003 subtest, and only on X with JDK11, from what I can see. The earliest failure is this May 8 run. Console log for one of the failures in that run:

03:32:06  Eclipse OpenJ9 VM (build master-bfaeffa, JRE 11 Linux amd64-64-Bit Compressed References 20240508_1838 (JIT enabled, AOT enabled)
03:32:06  OpenJ9   - bfaeffa
03:32:06  OMR      - 1bcf969
03:32:06  JCL      - b8f6121 based on jdk-11.0.24+1)

        Testing: rtc003
        Test start time: 2024/05/08 01:12:04 Pacific Standard Time
        Running command: "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation  -Xdump    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED -agentlib:jvmtitest=test:rtc003 -cp "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar:/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/../../testDependency/lib/asm-all.jar" com.ibm.jvmti.tests.util.TestRunner
        Time spent starting: 6 milliseconds
        Time spent executing: 1422 milliseconds
        Test result: FAILED
        Output from test:
         [OUT] *** Testing [1/1]:	testRedefineImplementedInterface
         [ERR] Exception in thread "main" java.lang.NoClassDefFoundError: com.ibm.oti.util.Msg (initialization failure)
         [ERR] 	at java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:160)
         [ERR] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:865)
         [ERR] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:859)
         [ERR] 	at java.base/java.lang.Thread.uncaughtException(Thread.java:1363)
         [ERR] Caused by: java.lang.ClassCastException: K0340
         [ERR] 	at java.base/java.lang.invoke.HandleCache.getStaticCache(HandleCache.java:124)
         [ERR] 	at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:644)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:313)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:216)
         [ERR] 	at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.open(SystemModuleFinders.java:448)
         [ERR] 	at java.base/jdk.internal.loader.BuiltinClassLoader.findResourceAsStream(BuiltinClassLoader.java:434)
         [ERR] 	at java.base/jdk.internal.loader.BootLoader.findResourceAsStream(BootLoader.java:146)
         [ERR] 	at java.base/java.lang.Module.getResourceAsStream(Module.java:1590)
         [ERR] 	at java.base/com.ibm.oti.vm.MsgHelp.loadMessages(MsgHelp.java:185)
         [ERR] 	at java.base/com.ibm.oti.util.PriviAction.run(PriviAction.java:172)
         [ERR] 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:690)
         [ERR] 	at java.base/com.ibm.oti.util.Msg.<clinit>(Msg.java:63)
         [ERR] 	at java.base/java.lang.ClassCastException.<init>(ClassCastException.java:71)
         [ERR] 	at java.base/java.lang.invoke.HandleCache.getStaticCache(HandleCache.java:124)
         [ERR] 	at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:644)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:313)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:216)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003$TestFunction.apply(rtc003.java:49)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003$TestFunction.apply(rtc003.java:47)
         [ERR] 	at java.base/java.util.Map.computeIfAbsent(Map.java:1003)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003.testRedefineImplementedInterface(rtc003.java:38)
         [ERR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         [ERR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         [ERR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         [ERR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:572)
         [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
         [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
         [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
         [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
        >> Success condition was not found: [Return code: 0]

A 25x grinder without JITServer did not result in any failures, and this is reproducible consistently locally with JITServer enabled.

These are the differences between failing May 8 build and the successful May 7 build:

• https://github.com/eclipse-openj9/openj9/compare/b05b2c1f6195fc312a4ba6889efb6d706570a8ee...bfaeffa721246eeed84345ce1dea45ad264385d7
• https://github.com/eclipse/omr/compare/acd6f7e0b0ce384cb2e941729635c91a3a1c451a...1bcf9698259cfdaee01be6215f6f9dcd04ca91c5
• https://github.com/adoptium/aqa-tests/compare/348e551a85654c638aac023d71b5a83d06e7c096...b2a0d5aeb2d2767524146e38ac59206e06d86871

I tested a build based on https://github.com/eclipse-openj9/openj9/commit/b05b2c1f6195fc312a4ba6889efb6d706570a8ee and https://github.com/eclipse/omr/commit/1bcf9698259cfdaee01be6215f6f9dcd04ca91c5 (so excluding bfaeffa721246eeed84345ce1dea45ad264385d7 in particular) and it did not fail, based on 3 test iterations. I tested a build based on bfaeffa721246eeed84345ce1dea45ad264385d7 and https://github.com/eclipse/omr/commit/acd6f7e0b0ce384cb2e941729635c91a3a1c451a, and 3 test iterations both failed. So, it may be https://github.com/eclipse-openj9/openj9/pull/19355 that is causing the failure, or at least causing it to appear consistently.