openj9 icon indicating copy to clipboard operation
openj9 copied to clipboard

Published 20 hours ago • verified publisher icon eclipse-openj9

Internal JITServer test failures

Open cjjdespres opened this issue 3 years ago • 162 comments

This thread collects various failures in internal JITServer tests that have been observed recently, for platforms other than Z. I'll be going through them to see if they can be reproduced without JITServer enabled.

Unable to reproduce

  • jdk_util_1 In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/271/ Comment here. The sub-test LoggingDeadlock2 failed with "Agent 7 timed out with a timeout of 960 seconds".

  • SharedClasses.SCM01.MultiCL_1 In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • MiniMix_5m_0 In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/, job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • DBBLoadTest_5m_0 In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • HCRLateAttachWorkload_previewEnabled_0 In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • ObjectTreeLoadTest_5m_1 In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • testOSMXBeanRemote_1, TestManagementAgent_0 In: job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/290/ Comment here. Tests fail to connect to remote server that was created.

  • LambdaLoadTest_CS_5m_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/ Comment here. Segfault.

  • MathLoadTest_bigdecimal_5m_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/ Comment here. "org.apache.logging.log4j.message.DefaultFlowMessageFactory not found"

  • ParallelStreamsLoadTest_J9_1, MauveMultiThrdLoad_5m_0, CpMp_MP_0, AutoMod1_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/ Comment here. Path/URI exceptions, string index out of bounds errors.

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_4 In: job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/278/ Comment here. "Exception in thread "Attach API initializer" java.lang.ExceptionInInitializerError".

  • testSCCMLTests1_openj9_1 In: job/Test_openjdk8_j9_sanity.functional_ppc64le_linux_jit_Personal/840/ Comment here. Various success and failure conditions not found.

  • cmdLineTester_SCHelperCompatTests_unix_0 In: job/Test_openjdk8_j9_sanity.functional_ppc64le_linux_jit_Personal/840/ Comment here. Various success and failure conditions not found.

  • LambdaLoadTest_special_J9_5m_9 In: job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/195/ Comment here. Segfault.

  • DaaLoadTest_daa2_CS_5m_0, MathLoadTest_autosimd_CS_5m_0, MathLoadTest_bigdecimal_CS_5m_0, CpMp_CpMp_0, SLTest_0, PatMod_PlatMod_0, PatMod_AppMod_0, PatMod_Unex_0, PatMod_Adv_0, PatModImg_AppMod_0, UpgModPath_Exp_0, UpgModPath_ExpImg_0, UpgModPath_JarImg_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here. Segfaults

  • DaaLoadTest_daa1_5m_1, LambdaLoadTest_J9_5m_1, AutoMod_Impl2_0, NioLoadTest_5m_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here. String index out of bounds errors.

  • MauveMultiThrdLoad_5m_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here. "Unable to parse --add-opens /: java.base/java.lang"

  • DaaLoadTest_daa3_CS_5m_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here, here. Segfault.

  • ParallelStreamsLoadTest_CS_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here, here. Segfault.

  • CpMp3_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here, here. Segfault.

  • AutoMod2_0 In: job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276 Comment here, here. Segfault.

  • cmdLineTester_classesdbgddrext_1 In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/ Console log here. The error encountered is JVMSHRC023E Cache does not exist.

  • testSCCacheManagement_0, cmdLineTester_classesdbgddrext_1, cmdLineTester_fastClassHashTable_0, cmdLineTester_javaAssertions_0, cmdLineTester_decompilationTests_nongold_3, cmdLineTester_lockWordAlignment_Object_Standard_0, cmdLineTester_lockWordAlignment_Object_iii_0, cmdLineTester_lockWordAlignment_Object_d_0, cmdLineTester_modularityddrtests17_1, cmdLineTester_shareClassesBadStackMap_0, testSCCMLTests3_1, testSCCMLTests4_1, testSCCMLTests6_1, testSCCMLSnapshot_1, testSCCMLAotMethodOperation_1, cmdLineTester_shrcdbgddrext_1, cmdLineTest_stackSizeInfoTest_0, cmdLineTester_verbosetest_5, cmdLineTester_vmRuntimeState_0, cmdLineTester_xlogTests_0, StackWalkerTestJava10_0, JCL_TEST_Java-Lang-Invoke_0, TestAttachAPIEnabling_0, TestSunAttachClasses_0, gcNotificationTest_Optthruput_1, gcNotificationTest_OptAvgpause_1, gcNotificationTest_Balanced_0, gcNotificationTest_Balanced_1, gcNotificationTest_Metronome_1, ContendedFieldsTests_90_0, threadMXBeanTestSuite2_2, threadMXBeanTimedParkTest_2, threadMXBeanTimedParkTest_3, memoryCategories_0, HashConsistency_0, jniOnLoadExceptions_1, badUTF8inJNI_1, badUTF8inJNI_2, testvmcheck_3, testvmcheck_6, TestSoftMxCallBackOOM_1, TestSoftMxCallBackExtend_1, MethodVisibilityTests_0, UnreflectTests_0 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/ Comment here. Segfaults.

  • SecurityTests_0, CDSAdaptorTest_0, cmdLineTester_bootStrapStaticVerify_0, cmdLineTester_EnableAssertionStatusTest_0, cmdLineTest_J9test_extended_0, cmdLineTest_J9test_consoleUpTo17_0, cmdLineTester_dumpromclasstests_0, cmdLineTester_getCallerClassTests_SE80_0, cmdLineTester_ignoreunrecognizedvmoptions_0, cmdLineTester_jep178_staticLinking_SE80_0, cmdLineTester_fieldwatchtests_0 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/ Comment here. Segfaults.

  • DaaLoadTest_all_5m_1, SharedClasses.SCM01.SingleCL_1, SharedClasses.SCM01.MultiThread_1, SharedClasses.SCM23.MultiThreadMultiCL_1 In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/ Comment here. Segfaults.

  • SecurityTests_0, cmdLineTester_dumpromclasstests_0, cmdLineTester_locales_0, cmdLineTester_lockWordAlignment_Object_i_0, testDDRExtJunit_MonitorsAndDeadlock2_1, testDDRExtJunit_MonitorsAndDeadlock3_1, testDDRExtJunit_MonitorsAndDeadlock6_1, TestAttachAPI_0, TestAttachErrorHandling_0, gcNotificationTest_GenCon_0, threadMXBeanTimersTest_3, HashPerformance_0, ContendedFieldsTests_90_1, ContendedFieldsTests_90_3 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/ Comment here. Null pointer exceptions.

  • StackWalkerTest_1 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/ Comment here. "Exception in thread "Attach API initializer" java.lang.InternalError: not invoker: invokeExact"

  • MathLoadTest_all_5m_0 In: job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/ Comment here. "java.lang.NoClassDefFoundError: org.apache.logging.log4j.message.DefaultFlowMessageFactory"

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/ Comment here. "Error: Could not find or load main class MainTester"

  • DaaLoadTest_daa1_special_5m_17, DaaLoadTest_daa3_special_5m_16, LambdaLoadTest_special_J9_5m_16, MathLoadTest_all_special_5m_16, MathLoadTest_all_special_5m_21, MathLoadTest_autosimd_special_5m_16, ClassLoadingTest_special_5m_15, ClassLoadingTest_special_5m_22, In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/ Comment here. Segfaults.

  • DaaLoadTest_daa1_special_5m_18, DaaLoadTest_daa1_special_5m_19, DaaLoadTest_daa1_special_5m_25, DaaLoadTest_daa1_special_5m_26, DaaLoadTest_daa1_special_5m_27, DaaLoadTest_daa2_special_5m_14, DaaLoadTest_daa2_special_5m_19, DaaLoadTest_daa3_special_5m_14, DaaLoadTest_daa3_special_5m_19, DaaLoadTest_daa3_special_5m_25, DaaLoadTest_all_special_5m_23, DaaLoadTest_all_special_5m_26, LambdaLoadTest_special_J9_5m_25, ParallelStreamsLoadTest_special_J9_17, ParallelStreamsLoadTest_special_J9_18, MathLoadTest_all_special_5m_24, MathLoadTest_autosimd_special_5m_23, MathLoadTest_autosimd_special_5m_24, MathLoadTest_bigdecimal_special_5m_23, MathLoadTest_bigdecimal_special_5m_24, ClassLoadingTest_special_5m_18 In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/ Comment here. "Null property: java.home".

  • DaaLoadTest_daa2_special_5m_22, DaaLoadTest_all_special_5m_22, MathLoadTest_autosimd_special_5m_22, LambdaLoadTest_special_J9_5m_22, LambdaLoadTest_special_J9_5m_29, MathLoadTest_all_special_5m_28, ClassLoadingTest_special_5m_29 In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/ Comment here. Various path/URI errors, string index out of bounds errors.

  • DaaLoadTest_daa1_special_5m_21 In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/192/ Comment here. Segfault.

  • SC_Softmx_UpDown_1, SC_Softmx_Increase_1 In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here. Unexpected crash, no other detail in the log.

  • MiniMix_aot_5m_0, SC_Softmx_JitAot_Linux_1, ConcurrentLoadTest_5m_0, LockingLoadTest_0, UtilLoadTest_5m_0, SharedClasses.SCM23.SingleCL_1 In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here. Segfaults.

  • DaaLoadTest_all_CS_5m_0, MathLoadTest_all_CS_5m_0, SharedClasses.SCM01.SingleCL_1 In: job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/ Comment here. Null pointer exceptions.

  • MiniMix_aot_5m_0 In: job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/274/ Comment here. An exception - the relevant log line is too corrupted to tell what kind.

  • ParallelStreamsLoadTest_special_J9_13 In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/, job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/ Comment here, here. "Null property: java.home".

  • DaaLoadTest_daa3_special_5m_13, DaaLoadTest_daa3_special_5m_17, DaaLoadTest_all_special_5m_16, DaaLoadTest_all_special_5m_21, ParallelStreamsLoadTest_special_J9_12 In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/ Comment here. Segfaults.

  • DaaLoadTest_all_special_5m_29, DaaLoadTest_daa1_special_5m_21 In: job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/ Comment here. "org.apache.logging.log4j.message.DefaultFlowMessageFactory not found"

  • SyntheticGCWorkload_DoubleMap_J9_0 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • cmdLineTester_defaultLazySymbolResolution_1 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • cmdLineTester_hangTest_0 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/ Comment here, here. Segfaults. Unable to reproduce in Grinder.

  • jdk_lang_1 In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/280/ Comment here. Some sort of thrown exception during the test, java.lang.Throwable: x[20] =0.

  • testSCCacheManagement_0 In: `job/Test_openjdk17_j9_extended.functional_ppc64le_linux_jit_Personal/199/. Comment here. "Invalid number of cache found".

  • shrtest_linux_SE80_1 In: job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/918/ Comment here. "Failure to unlink directory". May be a system error.

  • jdk_lang_0 In: job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/310/ Comment here. Test time out.

  • AutoMod2_0, ClassLoadingTest_special_5m_27, ClassLoadingTest_special_5m_29, CpMpModJar_0, CpMp_MP_0, DaaLoadTest_all_special_5m_27, DaaLoadTest_all_special_5m_29, DaaLoadTest_daa1_special_5m_20, DaaLoadTest_daa1_special_5m_25, DaaLoadTest_daa2_5m_1, DaaLoadTest_daa2_special_5m_29, DaaLoadTest_daa3_CS_5m_0, DaaLoadTest_daa3_special_5m_26, DaaLoadTest_daa3_special_5m_28, InternalAPIs_0, Jlink_AddMLimitM_0, Jlink_ReqMod_0, LambdaLoadTest_J9_5m_1, MathLoadTest_all_special_5m_24, MathLoadTest_autosimd_special_5m_21, MathLoadTest_autosimd_special_5m_28, MathLoadTest_bigdecimal_5m_0, MathLoadTest_bigdecimal_CS_5m_0, MathLoadTest_bigdecimal_special_5m_24, MathLoadTest_bigdecimal_special_5m_26, MathLoadTest_bigdecimal_special_5m_28, MauveSingleInvocLoad_J9_5m_1, PatModImg_AppMod_0, PatMod_AppMod_0, SLTest_0, UpgModPath_Exp_0, UpgModPath_JarImg_0, jdk11_tier1_buffer_0, jdk11_tier1_cipher_0, jdk11_tier1_iso8859_0, jdk_foreign_0, jdk_lang_0, jdk_lang_VarHandleTest_j9_0, jdk_lang_native_0, jdk_math_0, jdk_util_0 In: job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/250/, job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/247/, job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/249/ Comments here, here, here. Assertion failure, ((sp <= endOfStack)), also manifesting as a segfault related to stack walking.

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_0 In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/317/ Comment here. A couple of segfaults.

  • jdk_util_0 In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/334/, job/Test_openjdk8_j9_sanity.openjdk_ppc64le_linux_jit_Personal/336/ Comment here. "java.lang.RuntimeException: Incorrect hashCode computation."

  • LambdaLoadTest_special_J9_5m_8 In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/366/ Comment here. A segfault.

  • testSCCMLSoftmx_1 In: /job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/423 Comment here. Test timeout.

  • LambdaLoadTest_J9_5m_0 In: /job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/321 Comment here. Assertion failure "block->next() == NULL".

  • jdk11_tier1_cipher_0, jdk11_tier1_cipher_1 In: /job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/335 Comment here. Possibly some kind of FIPS-related error, and also test output mismatches.

  • Jep389Tests_testClinkerFfi_UpCall_0 In: /job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/422 Comment here. An assertion failure.

  • jdk_util_0 In: /job/Test_openjdk8_j9_sanity.openjdk_ppc64le_linux_jit_Personal/573 Comment here. A deadlock occurred.

  • LambdaLoadTest_special_J9_5m_12. Segfault. Comment here.

Resolved

  • testDefaultDisclaimMemory_3 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/185/ Console log here. Crash - assertion failure. Appears to be a duplicate of another issue.

  • ParallelStreamsLoadTest_special_J9_11 In: job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/254/ Console log here. Crash - assertion failure; segmentation error. Reproduced in view/Test_grinder/job/Grinder/21194/ without JITAAS with a 10% failure rate.

  • testSoftMxLocal_LP4k_3,testSoftMxDisclaimMemory_LP4k_3,testDefaultDisclaimMemory_3 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/188/ Comment here. Crash - assertion failure; segmentation error. Appears to be a duplicate of another issue.

  • jdk_lang_0,jdk_lang_1 In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/186/, job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/265/, job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/180/ Console log here. ~The failure seems similar to that reported in another issue.~ Fixed by this commit.

  • cmdLineTester_pltest_j9sig_ext_0, testSCCMLTests2_0, testSCCMLTests4_0, testSCCMLTests4_1, cmdLineTester_shrcdbgddrext_1, testDDRExt_SharedClasses_0, testDDRExtJunit_StackMap_0, testDDRExtJunit_MonitorsAndDeadlock1_1, testDDRExtJunit_MonitorsAndDeadlock2_0, testDDRExtJunit_MonitorsAndDeadlock2_1, testDDRExtJunit_MonitorsAndDeadlock3_0, testDDRExtJunit_MonitorsAndDeadlock5_0, testDDRExtJunit_MonitorsAndDeadlock5_1, TestJcmd_0, testOpenJ9DiagnosticsMXBean_0 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/268/ Some logs here. All the failures seem to relate to the failure of a core dump to be created. Possibly all are related to #5310.

  • cmdLineTester_getPid_0 In: job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/191/, job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/185/ Console log here. Test failure [OUT] FAIL: RuntimeMXBean.getPID() returned 0 instead of 15990794. Duplicate of #13085.

  • SharedClassesAPI_1 In: job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/265/ Console log here. Test failure Incorrect number of caches found. Expected at least 4, found 2. Duplicate of #13239.

  • Various TestJlm* and Test IBMJlm* tests. In: job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/276/, job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/275/, job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/191/, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276, job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/203/ Console log here, and here. All are either a crash while attempting to connect to a VM, or a failure of a VM to start. Haven't attempted to reproduce, but Marius here says that they've been happening for a while, probably aren't JITServer related, and are low priority.

  • cmdLineTester_fieldwatchtests_0 In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/271/ Console log and comments here. Test failure Invalid number of field reports detected: 0. Expected: 5. I've reproduced the failure with JITAAS, but not without it. Tracked by #14633.

  • jdk_lang_1 In: job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/267/ Comment here. Segfault. Tracked by #14182, not JITServer-specific.

  • cmdLineTester_jvmtitests_debug_3, cmdLineTester_jvmtitests_debug_5, cmdLineTester_jvmtitests_debug_7, cmdLineTester_jvmtitests_debug_8, cmdLineTester_jvmtitests_debug_openj9_none_SCC_11, cmdLineTester_jython_6 In: job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/899/, job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/902/ Comment here, here. Segfaults. Reproduction: with JITAAS, job/Grinder/21630/, 418/918 failures; without JITAAS, job/Grinder/21638/, 0/1170 failures. Tracked by #14704.

  • cmdLineTester_jvmtitests_debug_10, cmdLineTester_jvmtitests_debug_11, cmdLineTester_jvmtitests_debug_openj9_none_SCC_3, cmdLineTester_jvmtitests_debug_openj9_none_SCC_5, cmdLineTester_jvmtitests_debug_openj9_none_SCC_7, cmdLineTester_jvmtitests_debug_openj9_none_SCC_8, cmdLineTester_jvmtitests_debug_openj9_none_SCC_10 In: job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/898/, job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/899/, job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/902/ Comment here, here. Segfaults. Each has been reproduced with JITAAS and none without (see comment for details). Tracked by #14704.

  • TestFileLocking_SE80_0 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/281/ Comment here. Segfault. Reproduced with 4/50 failure rate in job/Grinder/21553/ with JITAAS, unable to reproduce without it. Tracked by #14704.

  • testDefaultDisclaimMemory_2 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/281/, job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/285/ Comment here. Segfault. Reproduction: with JITAAS, job/Grinder/21631/, 26/195 failures); without JITAAS, job/Grinder/21633/, 0/195 failures. Tracked by #14704.

  • testSoftMxDisclaimMemory_2 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/289/ Comment here. Segfault. Reproduction: with JITAAS, job/Grinder/21673/, 6/195 failures; without JITAAS, job/Grinder/21674/, 0/195 failures. Tracked by #14704.

  • TestArrayCopy_3 In: job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/895/ Comment here. Segfault. Tracked by #14704.

  • jdk_util_0 In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/213/. Comment here. Segfault. Tracked by #14704.

  • MathLoadTest_autosimd_CS_5m_0 In job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/292/ Comment here. Test output did not match expected value. Reproduction: 1/80 failures in job/Grinder/21725/, 0/200 failures in job/Grinder/21737/. Tracked by #14755.

  • MathLoadTest_autosimd_special_5m_29 In: job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/213/ Comment here. Test output did not match expected value. Tracked by #14755.

  • jdk_foreign_0, jdk_foreign_1 In: job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/217/. Comment here. "Method TestUnsupportedPlatform.testNoInitialization()[pri:0, instance:TestUnsupportedPlatform@8fe597f0] should have thrown an exception of type class java.lang.ExceptionInInitializerError". Tracked by #14828.

  • MiniMix_3h_0 In: job/Test_openjdk8_j9_special.system_x86-64_linux_jit_Personal/304/ Comment here. "Inconsistency detected by ld.so". Probably a glibc issue.

  • NioLoadTest_5m_0 In: job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/232/ Comment here. Segmentation error. Appears to be #14704.

  • jdk_lang_0, jdk_util_0, jdk_util_1, MathLoadTest_all_CS_5m_0 In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/233/, job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/235/, job/Test_openjdk8_j9_sanity.openjdk_x86-64_linux_jit_Personal/306/, job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/313/ Comment here. Various math test failures (test output mismatches, stack overflow errors). Tracked by #14864.

  • DaaLoadTest_daa2_CS_5m_0 In: job/Test_openjdk8_j9_sanity.system_ppc64le_linux_jit_Personal/309/ Comment here. Segfault. Tracked by #10489.

  • testSCCMLAotMethodOperation_1 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/315/ Tracked by #14704.

  • testvmcheck_3 In: job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/316/ Tracked by #14704.

  • TestJps_0 In: job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/317/ "Missing argument: foo". Looks like it's tracked by https://github.com/eclipse-openj9/openj9/issues/11509.

  • threadMXBeanTimedParkTest_0, threadMXBeanTimedParkTest_1 In: job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/246/ Comment here. Assertion failure - wait time is higher than expected. Tracked by #14470.

  • jdk_lang_0 In: job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/237/ Comment here. "java/nio/charset/CoderMalfunctionError: java.lang.IllegalArgumentException: newPosition > limit". I think this may be an instance of #14654.

  • cmdLineTest_sigabrtHandlingTest_0 In: job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/256/ Comment here. A java core failed to be created. Tracked by #11979.

  • TestSoftMxCallBackOOM_1, JCL_Test_OutOfMemoryError_1 In: job/Test_openjdk17_j9_extended.functional_ppc64le_linux_jit_Personal/280, job/Test_openjdk11_j9_sanity.functional_ppc64le_linux_jit_Personal/939/ Comment here. Assertion failure. Fixed after the reversion of #15201.

  • testRASAPI_0 In: job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/367/ Comment here. Dump couldn't be written to file. Tracked by #11800.

  • cmdLineTester_GCCheck_1, cmdLineTester_GCCheck_2, cmdLineTester_classesdbgddrext_0, cmdLineTester_modularityddrtests11_0, cmdLineTester_shrcdbgddrext_1, testRASAPI_0 In: job/Test_openjdk8_j9_sanity.functional_x86-64_linux_jit_Personal/983/, job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/389/ Comment here. Seems to be some kind of test setup failure.

  • DaaLoadTest_daa1_special_5m_20 In: /job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/300/ Comment here. Possible hang detected. Tracked by #13477.

  • jdk_lang_1 In: job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/376/ Comment here. Segfault. Tracked by #15321.

  • ConcurrentLoadTest_5m_1 In: job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/375/ Comment here. Tracked by #8493.

  • ClassLoadingTest_CS_5m_0, DaaLoadTest_all_CS_5m_0, DaaLoadTest_daa1_CS_5m_0, DaaLoadTest_daa2_CS_5m_0, DaaLoadTest_daa3_CS_5m_0, HCRLateAttachWorkload_0, HCRLateAttachWorkload_1, HCRLateAttachWorkload_previewEnabled_0, HCRLateAttachWorkload_previewEnabled_1, LambdaLoadTest_CS_5m_0, MathLoadTest_all_CS_5m_0, MathLoadTest_autosimd_CS_5m_0, MathLoadTest_bigdecimal_CS_5m_0, ParallelStreamsLoadTest_CS_0 In: /job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/386, /job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/307/, /job/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal/384/, /job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/379/, /job/Test_openjdk8_j9_sanity.system_ppc64le_linux_jit_Personal/380/, /job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/293/, /job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/293/ Comment here. Assertion failed: _methodInfo->getMethodInfo() == reinterpret_cast<TR_OpaqueMethodBlock *>(_j9method). Tracked by #15426.

  • cmdLineTest_J9test_common_0 In: job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/325 Comment here. Segfault. Tracked by #13577.

  • LambdaLoadTest_special_J9_5m_21 In: job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/405/ Comment here. Segfault. Tracked by #13617.

  • ClassLoadingTest_special_5m_18, ClassLoadingTest_special_5m_19, ClassLoadingTest_special_5m_19 , ClassLoadingTest_special_5m_24, DaaLoadTest_all_special_5m_18, DaaLoadTest_all_special_5m_19, DaaLoadTest_all_special_5m_24, DaaLoadTest_daa1_special_5m_18, DaaLoadTest_daa1_special_5m_19, DaaLoadTest_daa1_special_5m_24, DaaLoadTest_daa2_special_5m_18, DaaLoadTest_daa2_special_5m_19, DaaLoadTest_daa2_special_5m_23, DaaLoadTest_daa2_special_5m_24, DaaLoadTest_daa3_special_5m_24, LambdaLoadTest_special_J9_5m_18, LambdaLoadTest_special_J9_5m_24, MathLoadTest_all_special_5m_18, MathLoadTest_all_special_5m_19, MathLoadTest_all_special_5m_24, MathLoadTest_autosimd_special_5m_18, MathLoadTest_autosimd_special_5m_19, MathLoadTest_bigdecimal_special_5m_18, MathLoadTest_bigdecimal_special_5m_19, MathLoadTest_bigdecimal_special_5m_24, ParallelStreamsLoadTest_special_J9_18 In: /job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/411, /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/318 Comment here. Segfault. Tracked by #15601.

  • DaaLoadTest_all_special_5m_27, DaaLoadTest_daa1_special_5m_26, LambdaLoadTest_special_J9_5m_25, MathLoadTest_all_special_5m_25, MathLoadTest_all_special_5m_27, MathLoadTest_autosimd_special_5m_23, MathLoadTest_bigdecimal_special_5m_25, MathLoadTest_bigdecimal_special_5m_26, MauveMultiThrdLoad_special_5m_13, MauveMultiThrdLoad_special_5m_24, MauveMultiThrdLoad_special_5m_25 In: /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/341, /job/Test_openjdk8_j9_special.system_ppc64le_linux_jit_Personal/421, /job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/356, /job/Test_openjdk8_j9_special.system_ppc64le_linux_jit_Personal/420, /job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/340, /job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/421 Comment here. Memory corruption/malloc errors. Tracked by #15751.

  • SharedClasses.SCM23.MultiThreadMultiCL_0, SharedClasses.SCM23.MultiThread_0, jdk_util_0 In: /job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/376, /job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/448, /job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/374 Comment here. Segfault. Tracked by #15852.

  • cmdLineTest_J9test_common_0 In: /job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/386 Comment here. Segfault. Tracked by #13577.

  • jdk_util_1 In: /job/Test_openjdk8_j9_sanity.openjdk_x86-64_linux_jit_Personal/475, /job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/478 Comment here. Assertion failed, "guardOkForExpr: should not intern OSR guard n3908n". Tracked by #16115.

  • DaaLoadTest_daa1_CS_5m_0, DaaLoadTest_daa3_CS_5m_0, LambdaLoadTest_CS_5m_0, MathLoadTest_bigdecimal_CS_5m_0, DaaLoadTest_daa2_CS_5m_0, ParallelStreamsLoadTest_CS_0, MathLoadTest_autosimd_CS_5m_0, ClassLoadingTest_CS_5m_0 , DaaLoadTest_all_CS_5m_0, MathLoadTest_all_CS_5m_0 In: /job/Test_openjdk11_j9_extended.system_x86-64_linux_jit_Personal/488, /job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/415, /job/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal/484, /job/Test_openjdk11_j9_sanity.system_x86-64_linux_jit_Personal/491, /job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/416, /job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/401, /job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/484, /job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/486, /job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/401, /job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/487 Comment here. Segfault. Tracked by #16149.

  • TestJcmd_0, threadMXBeanTestSuite1_2, threadMXBeanTestSuite1_3, threadMXBeanTestSuite1_5, threadMXBeanTestSuite2_0, threadMXBeanTestSuite2_2, threadMXBeanTestSuite2_3, testJCMMXBeanRemote_0, threadMXBeanTestSuite1_0, threadMXBeanTestSuite1_1, threadMXBeanTestSuite1_4, threadMXBeanTestSuite1_6, threadMXBeanTestSuite1_7, threadMXBeanTestSuite2_1, testJCMMXBeanRemote_1, TestJlmLocal_0, TestJlmLocal_1 In: /job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/426, /job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/506, /job/Test_openjdk11_j9_sanity.system_x86-64_linux_jit_Personal/494, /job/Test_openjdk17_j9_sanity.system_x86-64_linux_jit_Personal/419 Comment here. Test failures related to threading. Has been resolved.

  • cmdLineTester_criu_random_0 In: /job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/480/, among others Comment here. The initial checkpoint does not seem to be created in the first place. Tracked by https://github.com/eclipse-openj9/openj9/issues/16502.

  • jdk_lang_j9, on ppc64le. Null pointer exception in SealedClassesReflectionTest. Comment here. Tracked by https://github.com/eclipse-openj9/openj9/issues/17254.

cjjdespres avatar Feb 17 '22 14:02 cjjdespres

From build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/185/, failure in testDefaultDisclaimMemory_3, with console output

testDefaultDisclaimMemory_3 Start Time: Sat Feb 12 02:28:27 2022 Epoch Time (ms): 1644661707905
variation: Mode501
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
[IncludeExcludeTestAnnotationTransformer] [INFO] EXCLUDE_FILE environment variable: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_17.txt,/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
[IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_17.txt
[IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
...
... TestNG 6.14.2 by Cédric Beust ([email protected])
...

Invalid class in objectPtr=00007F82A63115A0
10:28:29.463 0x7f823c003a00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK17_x86-64_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:1258: ((false))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/12 02:28:29 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/core.20220212.022829.32526.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/core.20220212.022829.32526.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/javacore.20220212.022829.32526.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/javacore.20220212.022829.32526.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/Snap.20220212.022829.32526.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/Snap.20220212.022829.32526.0003.trc
JVMDUMP013I Processed dump event "traceassert", detail "".

testDefaultDisclaimMemory_3_FAILED

Seems to be related to https://github.com/eclipse-openj9/openj9/issues/14396, and shows up without JITServer enabled.

cjjdespres avatar Feb 17 '22 14:02 cjjdespres

From build job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/, failure in cmdLineTester_classesdbgddrext_1, with excerpted console output

 ===============================================
 Running test cmdLineTester_classesdbgddrext_1 ...
 ===============================================
 cmdLineTester_classesdbgddrext_1 Start Time: Sun Feb 13 01:17:51 2022 Epoch Time (ms): 1644743871414
 variation: Mode610
 JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon
 Start SysV clean up for user:all
 Clean up is finished:
 	Shared Memory Segments Removed:0
 	Shared Semaphores Removed:1
 Parsing exclude list...
   add excludes for [all]
   add excludes for [linux_ppc-64]
 DONE in 0 ms.

 No tests are excluded.

 *** Starting test suite: J9 shared cache dbgext Tests ***
 Testing: Create new shared cache with partition
 Test start time: 2022/02/13 01:17:52 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xaot:forceaot,count=0,disableAsyncCompilation -Xshareclasses:name=shareddbgext,reset,modified=mod1
 Time spent starting: 90 milliseconds
 Time spent executing: 10087 milliseconds
 Test result: PASSED

 Executing command: rm -f j9core.dmp

 Testing: Create core file
 Test start time: 2022/02/13 01:18:02 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xmx4m -Xaot:forceaot,count=0 -Xshareclasses:name=shareddbgext -cp /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/utils/utils.jar -Xdump:system:file=j9core.dmp,events=vmstop org.openj9.test.ivj.Hanoi 2
 Time spent starting: 8 milliseconds
 Time spent executing: 13650 milliseconds
 Test result: PASSED

 Testing: Remove shared cache
 Test start time: 2022/02/13 01:18:16 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=shareddbgext,destroy
 Time spent starting: 19 milliseconds
 Time spent executing: 106 milliseconds
 Test result: FAILED
  [ERR] JVMSHRC023E Cache does not exist
  [ERR] JVMSHRC023E Cache does not exist
 >> Success condition was not found: [Output match: destroyed]

I haven't been able to reproduce this with the grinders that I've run.

cjjdespres avatar Feb 17 '22 14:02 cjjdespres

From build job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/254/, failure in ParallelStreamsLoadTest_special_J9_11 with console output

===============================================
Running test ParallelStreamsLoadTest_special_J9_11 ...
===============================================
ParallelStreamsLoadTest_special_J9_11 Start Time: Sun Feb 13 03:52:10 2022 Epoch Time (ms): 1644753130302
variation: Mode501
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
STF 03:52:10.589 - =========================   S T F   =========================
systemtest-prereqs has been processed, and set to: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqsRetrieving amount of free space on drive containing /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11
There is 204283 Mb free
STF 03:52:10.607 - ==================   G E N E R A T I O N   ==================
STF 03:52:10.609 - Checking JVM: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image
STF 03:52:10.610 - Starting process to generate scripts: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java  -Dlog4j.skipJansi=true -Djava.system.class.loader=net.adoptopenjdk.stf.runner.StfClassLoader -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/system/lambdaLoadTest/..//systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/system/lambdaLoadTest/..//systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/scripts/../bin net.adoptopenjdk.stf.runner.StfRunner -properties "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/stf_parameters.properties, , /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/config/stf.properties" -testDir "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest"
GEN 03:52:13.980 - Found test. Project: 'openjdk.test.load' class: 'ParallelStreamsLoadTest.class' Dir: '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.load/bin'
GEN 03:52:14.053 - Found test. Project: 'openjdk.test.load' class: 'net.adoptopenjdk.stf.ParallelStreamsLoadTest'
GEN Classpath directories used by project 'openjdk.test.load':
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.load/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.load/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/junit.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar
GEN 03:52:14.606 - Using Mode NoOptions. Values = ''
GEN 03:52:14.750 -
GEN 03:52:14.750 - Test command summary:
GEN 03:52:14.751 -   Step  Stage   Command           Description
GEN 03:52:14.752 -  -----+--------+-----------------+------------
GEN 03:52:14.752 -     1  execute  Run java          Run parallel streams load test
STF 03:52:14.803 -
STF 03:52:14.803 - Script generation completed
STF 03:52:14.803 -
STF 03:52:14.805 -
STF 03:52:14.805 - =======================   S E T U P   =======================
STF 03:52:14.805 - Running setup: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/setUp.pl
STF 03:52:14.932 - SETUP stage completed
STF 03:52:14.937 -
STF 03:52:14.937 - ====================   E X E C U T E -   ====================
STF 03:52:14.937 - Running execute: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/execute.pl
STF 03:52:15.055 -
STF 03:52:15.055 - Java version
STF 03:52:15.055 - Running: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java -version
openjdk version "11.0.15-internal" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15-internal+0-adhoc.jenkins.BuildJDK11ppc64lelinuxjitPersonal)
Eclipse OpenJ9 VM (build master-c462b8b9d0c, JRE 11 Linux ppc64le-64-Bit Compressed References 20220213_936 (JIT enabled, AOT enabled)
OpenJ9   - c462b8b9d0c
OMR      - ae234b55240
JCL      - 24e7209ed42 based on jdk-11.0.15+1)
STF 03:52:15.311 -
STF 03:52:15.312 - +------ Step 1 - Run parallel streams load test
STF 03:52:15.312 - | Run foreground process
STF 03:52:15.312 - |   Program:     /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
STF 03:52:15.312 - |   Mnemonic:    LT
STF 03:52:15.312 - |   Echo:        ECHO_ON
STF 03:52:15.312 - |   Expectation: CLEAN_RUN within 1h
STF 03:52:15.312 - |
STF 03:52:15.315 - Running command: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java -Xmx1024M -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.load/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/junit.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.lambdasAndStreams/bin net.adoptopenjdk.loadTest.LoadTest -resultsDir /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results -resultsPrefix 1.LT. -inactivityLimit 60m -reportFailureLimit 1 -abortAtFailureLimit 10 -maxTotalLogFileSpace 200M -maxSingleLogSize 1/25 -suite.parallelStreams.threadCount 2 -suite.parallelStreams.inventoryFile /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml -suite.parallelStreams.inventoryExcludeFile none -suite.parallelStreams.totalNumberTests 2 -suite.parallelStreams.selection sequential -suite.parallelStreams.seed -1 -suite.parallelStreams.repeatCount 1 -suite.parallelStreams.thinkingTime 0ms..0ms
STF 03:52:15.315 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.stderr
STF 03:52:15.315 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.stdout
STF 03:52:15.337 - Monitoring processes: LT
LT  03:52:18.683 - Load test parameters
LT  03:52:18.692 -   Time limited         = false
LT  03:52:18.693 -   abortIfOutOfMemory   = true
LT  03:52:18.694 -   reportFailureLimit   = 1
LT  03:52:18.695 -   abortAtFailureLimit  = 10
LT  03:52:18.698 -   maxTotalLogFileSpace = 209715200
LT  03:52:18.699 -   maxSingleLogSize     = 8388608
LT  03:52:18.700 - Parameters for suite 0
LT  03:52:18.702 -   Suite name     = parallelStreams
LT  03:52:18.703 -   Number threads = 2
LT  03:52:18.704 -   Supplied seed  = -1
LT  03:52:18.705 -   Inventory file = /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml
LT  03:52:18.706 -   Exclude file   = none
LT  03:52:18.707 -   Number tests   = 2
LT  03:52:18.708 -   Repeat count   = 1
LT  03:52:18.709 -   Thinking time  = 0ms..0ms
LT  03:52:18.711 -   Selection mode = sequential
LT  03:52:18.726 - Parsing inventory file. Root=/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory File=openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml
LT  03:52:18.740 - Final test list:
LT  03:52:18.745 -   0 JUnit[net.adoptopenjdk.test.streams.TestParallelStreamOperations]  Weighting=1
LT  03:52:18.899 - Starting thread. Suite=0 thread=0
LT  03:52:18.902 - Starting thread. Suite=0 thread=1
LT  03:52:38.936 - Completed 100.0%. Number of tests started=2
LT  03:52:58.971 - Completed 100.0%. Number of tests started=2 (+0)
LT  stderr 11:53:18.527 0x7ff830002400    j9mm.107    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3577: ((false && (isObjectInEvacuateMemory(object))))
LT  stderr 11:53:18.527 0x7ff83c002400    j9mm.107    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3577: ((false && (isObjectInEvacuateMemory(object))))
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/13 03:53:18 - please wait.
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/13 03:53:18 - please wait.
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp' in response to an event
LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.3133622.
LT  stderr
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt' in response to an event
STF 03:53:22.925 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
STF 03:53:22.926 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp' in response to an event
LT  stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.3133632.
LT  stderr
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt' in response to an event
LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc' in response to an event
STF 03:53:31.952 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt
STF 03:53:31.953 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
STF 03:53:31.953 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
LT  stderr Snap file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr JVMDUMP013I Processed dump event "traceassert", detail "".
STF 03:53:38.476 - **FAILED** Process LT  ended with exit code (255) and not the expected exit code/s (0)
STF 03:53:38.476 - Monitoring Report Summary:
STF 03:53:38.476 -   o Process LT  has crashed unexpectedly
STF 03:53:38.477 - Killing processes: LT
STF 03:53:38.477 -   o Process LT  pid 3133542 is not running
**FAILED** at step 1 (Run parallel streams load test). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/execute.pl line 97.
STF 03:53:38.904 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 03:53:38.904 -
STF 03:53:38.904 - ====================   T E A R D O W N   ====================
STF 03:53:38.904 - Running teardown: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/tearDown.pl
STF 03:53:39.027 - TEARDOWN stage completed
STF 03:53:39.030 -
STF 03:53:39.030 - =====================   R E S U L T S   =====================
STF 03:53:39.030 - Stage results:
STF 03:53:39.030 -   setUp:     pass
STF 03:53:39.030 -   execute:  *fail*
STF 03:53:39.030 -   teardown:  pass
STF 03:53:39.031 -
STF 03:53:39.031 - Overall result: **FAILED**

ParallelStreamsLoadTest_special_J9_11_FAILED

I haven't been able to reproduce this with the grinders that I've run.

cjjdespres avatar Feb 17 '22 14:02 cjjdespres

From build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/188/, failures in testSoftMxLocal_LP4k_3,testSoftMxDisclaimMemory_LP4k_3,testDefaultDisclaimMemory_3. I think this one is a duplicate of https://github.com/eclipse-openj9/openj9/issues/14396 as well, based on that thread.

cjjdespres avatar Feb 17 '22 14:02 cjjdespres

From build job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/186/, failures in jdk_lang_0 and jdk_lang_1. I believe the relevant console output is

03:51:13  java.lang.Throwable
03:51:13  	at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13  	at Basic.fail(Basic.java:2817)
03:51:13  	at Basic.fail(Basic.java:2818)
03:51:13  	at Basic.equal(Basic.java:2824)
03:51:13  	at Basic.realMain(Basic.java:1385)
03:51:13  	at Basic.main(Basic.java:2827)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  >'=C:=\,OPENJ9_JAVA_COMMAND_LINE=/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/openjdkbinary/j2sdk-image/bin/java -XX:+DisplayVMOutputToStderr -classpath /home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/TKG/output_16450868001065/jdk_lang_1/work/classes/1/java/lang/ProcessBuilder/Basic_id1.d:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/ProcessBuilder:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/TKG/output_16450868001065/jdk_lang_1/work/classes/1/test/lib:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/jvmtest/openjdk/jtreg/lib/jtreg.jar Basic$JavaChild System.getenv(),'< not equal to '=C:=\,'
03:51:13  java.lang.Throwable
03:51:13  	at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13  	at Basic.fail(Basic.java:2817)
03:51:13  	at Basic.fail(Basic.java:2818)
03:51:13  	at Basic.equal(Basic.java:2824)
03:51:13  	at Basic.realMain(Basic.java:1902)
03:51:13  	at Basic.main(Basic.java:2827)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  Incorrect handling of envstrings containing NULs
03:51:13  java.lang.Throwable
03:51:13  	at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13  	at Basic.fail(Basic.java:2817)
03:51:13  	at Basic.fail(Basic.java:2818)
03:51:13  	at Basic.check(Basic.java:2821)
03:51:13  	at Basic.realMain(Basic.java:1956)
03:51:13  	at Basic.main(Basic.java:2827)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  java.lang.AssertionError: Some tests failed
03:51:13  	at Basic.main(Basic.java:2829)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13  	at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  
03:51:13  JavaTest Message: Test threw exception: java.lang.AssertionError: Some tests failed
03:51:13  JavaTest Message: shutting down test
03:51:13  
03:51:13  STATUS:Failed.`main' threw exception: java.lang.AssertionError: Some tests failed

cjjdespres avatar Feb 17 '22 15:02 cjjdespres

Builds job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/265/ and job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/180/ fail in exactly the same way that https://github.com/eclipse-openj9/openj9/issues/14538#issuecomment-1043050057 does.

cjjdespres avatar Feb 17 '22 15:02 cjjdespres

For easier tracking I suggest we have a list in the description of this issue with some brief comments about what we've learned. E.g. fails without JITServer too, already tracked by issue N, crash or not, etc. I am suggesting this because this issue is going to become very large, very soon :-)

mpirvu avatar Feb 17 '22 15:02 mpirvu

Definitely a good idea.

cjjdespres avatar Feb 17 '22 16:02 cjjdespres

The recent java/lang/ProcessBuilder/Basic.java failures are caused by changes merged yesterday and I expect they will be fixed by changes to the OpenJDK repos within the next hour. Fixes are under review now. https://github.com/eclipse-openj9/openj9/pull/14520 https://github.com/eclipse-openj9/openj9/pull/14528

pshipton avatar Feb 17 '22 17:02 pshipton

The fixes are merged now.

pshipton avatar Feb 17 '22 17:02 pshipton

The grinder https://hyc-runtimes-jenkins.swg-devops.com/view/Test_grinder/job/Grinder/21194/ reproduces the ParallelStreamsLoadTest_special_J9_11 failure without JITAAS.

cjjdespres avatar Feb 17 '22 20:02 cjjdespres

The build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/268/ has failures in:

  • cmdLineTester_pltest_j9sig_ext_0
  • testSCCMLTests2_0
  • testSCCMLTests4_0
  • testSCCMLTests4_1
  • cmdLineTester_shrcdbgddrext_1
  • testDDRExt_SharedClasses_0
  • testDDRExtJunit_StackMap_0
  • testDDRExtJunit_MonitorsAndDeadlock1_1
  • testDDRExtJunit_MonitorsAndDeadlock2_0
  • testDDRExtJunit_MonitorsAndDeadlock2_1
  • testDDRExtJunit_MonitorsAndDeadlock3_0
  • testDDRExtJunit_MonitorsAndDeadlock5_0
  • testDDRExtJunit_MonitorsAndDeadlock5_1
  • TestJcmd_0
  • testOpenJ9DiagnosticsMXBean_0

I believe that testSCCMLTests2_0, testSCCMLTests4_0, testSCCMLTests4_1 are all being tracked by #5310. They fail because certain core files aren't created that should be. The rest all seem to fail in ways vaguely similar to those three tests. An excerpt from the (very large) output of cmdLineTester_pltest_j9sig_ext_0:

 [ERR] ----------------------------------------
 [ERR] dump tests
 [ERR] ----------------------------------------
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_verify_functiontable_slots
 [ERR] Ending test j9dump_verify_functiontable_slots
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_with_name
 [ERR]   calling j9dump_create with filename: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/j9dump_test_create_dump_with_name
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  213: j9dump_test_create_dump_with_name j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252389 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/core"
 [ERR] 
 [ERR] 		LastErrorNumber: -108
 [ERR] 		LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_with_name
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_from_signal_handler
 [ERR]   calling j9dump_create with filename: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/j9dump_test_create_dump_from_signal_handler
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  311: j9dump_test_create_dump_from_signal_handler j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252395 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/core"
 [ERR] 		LastErrorNumber: -108
 [ERR] 		LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_from_signal_handler
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_with_NO_name
 [ERR]   calling j9dump_create with empty filename
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  153: j9dump_test_create_dump_with_NO_name j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252401 was not found. Expected to find core file with name "core"
 [ERR] 		LastErrorNumber: -108
 [ERR] 		LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_with_NO_name
 [ERR] 
 [ERR] Dump test done, failures detected.

And an excerpt from testDDRExtJunit_MonitorsAndDeadlock5_1:

     [exec] JVMDUMP039I Processing dump event "throw", detail "j9vm/test/corehelper/HelperExceptionForCoreGeneration" at 2022/02/18 02:07:24 - please wait.
     [exec] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP' in response to an event
     [exec] JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.1275170.
     [exec] 
     [exec] HelperExceptionForCoreGeneration is thrown and caught successfuly.
     [exec] Forcefully terminating process.
     [exec] addTestJitHint adding hint to openj9/internal/tools/attach/target/AttachHandler$1.<clinit>
     [exec] addTestJitHint adding hint to java/util/Vector$Itr.<init>
     [exec] addTestJitHint adding hint to java/lang/Thread$State.values
     [exec] JVMDUMP012E Error in System dump: The core file created by child process with pid = 1275170 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/core"
     [exec] JVMDUMP013I Processed dump event "throw", detail "j9vm/test/corehelper/HelperExceptionForCoreGeneration".

TCK.destroy.cache:
     [echo] Destroying cache
     [echo] Running j9vm.test.corehelper.StackMapCoreGenerator to destroy the cache
     [echo] Using JVM : /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
     [echo] classname = "j9vm.test.corehelper.StackMapCoreGenerator"
     [echo] Java VM Args:
     [echo]   jvmarg = -Xshareclasses:name=ddrextjunitSCC,destroy
     [java] JVMSHRC806I Compressed references persistent shared cache "ddrextjunitSCC" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
     [java] Java Result: 1

TCK.run.tests.ddrext:
     [echo] Running the DDR Extension Test
     [java] openjdk version "1.8.0_332-internal"
     [java] OpenJDK Runtime Environment (build 1.8.0_332-internal-_2022_02_18_08_00-b00)
     [java] Eclipse OpenJ9 VM (build master-c0f5771, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20220218_965 (JIT enabled, AOT enabled)
     [java] OpenJ9   - c0f5771
     [java] OMR      - 64690f6
     [java] JCL      - bd2e6cf based on jdk8u332-b02)
     [java] 
     [java] [AutoRun] [INFO] Warning: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP is not directory or file!
     [java] [AutoRun] [ERROR] No dump file under /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP

BUILD FAILED
/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/tck_ddrext.xml:152: The following error occurred while executing this line:
/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/tck_ddrext.xml:124: Java returned: 255

The only one that's a little different is cmdLineTester_shrcdbgddrext_1. That one starts with the failure:

Testing: Check whether dump file exist
Test start time: 2022/02/18 01:56:32 Pacific Standard Time
Running command: ls
Time spent starting: 3 milliseconds
Time spent executing: 18 milliseconds
Test result: FAILED
Output from test:
 [OUT] Calculation.class
 [OUT] Calculation.java
 [OUT] cmdlinetestresources.jar
 [OUT] com
 [OUT] Dump.class
 [OUT] Dump.java
 [OUT] j9vm
 [OUT] JavaRuntimeVersion.class
 [OUT] JavaRuntimeVersion.java
 [OUT] META-INF
 [OUT] SystemProperties.class
 [OUT] SystemProperties.java
 [OUT] VMBench
 [OUT] WithDebugInfo.class
 [OUT] WithDebugInfo.java
 [OUT] WithoutDebugInfo.class
 [OUT] WithoutDebugInfo.java
>> Success condition was not found: [Output match: j9core1.dmp]

and the subsequent tests fail because j9core1.dmp doesn't exist. But even here it's a problem with the creation of a core dump that causes the tests to fail.

In #5310 it is suspected that the failures are caused by machine issues. Perhaps that is the case here as well. The child build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/259/ had all the failures, running on computer ub20x64rt1-1.

cjjdespres avatar Feb 18 '22 15:02 cjjdespres

Related to these failures, I ran a subset of the failing tests a few times on the same machine in this run and got all failures, and ran the same tests on a few different automatically-allocated machines in this run and got all successes. So it does seem like a problem with ub20x64rt1-1.

cjjdespres avatar Feb 18 '22 18:02 cjjdespres

I tried to reproduce the cmdLineTester_classesdbgddrext_1 failure from job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/ again, including running on the same machine, and I still can't reproduce it. I think it may just be highly intermittent.

cjjdespres avatar Feb 18 '22 20:02 cjjdespres

From job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/191/, failure in cmdLineTester_getPid_0, with output

cmdLineTester_getPid_0 Start Time: Sat Feb 19 02:32:37 2022 Epoch Time (ms): 1645266757691
variation: NoOptions
JVM_OPTIONS: -XX:+UseJITServer 
*** Starting test suite: getPid test ***
Testing: Compare RuntimeMXBean.getPid() with getting PID from system
Test start time: 2022/02/19 02:32:48 Pacific Standard Time
Running command: perl /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/runtimemxbeanTests/getPidTest.pl "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer  -cp /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/runtimemxbeanTests/runtimemxbeanTests.jar GetPid
Time spent starting: 88 milliseconds
Time spent executing: 5059 milliseconds
Test result: FAILED
Output from test:
 [OUT] FAIL: RuntimeMXBean.getPID() returned 0 instead of 31948
>> Success condition was not found: [Output match: PASS]
>> Failure condition was found: [Output match: FAIL]

Appears to be a duplicate of #13085.

Build job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/185/ experienced exactly the same failure.

cjjdespres avatar Feb 21 '22 14:02 cjjdespres

From job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/269/, failure in testSoftMxDisclaimMemory_3. In addition, the build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/193/ has a failure in testSoftMxDisclaimMemory_LP4k_3. Both have been experienced before and are being tracked elsewhere. I suppose I'll not report failures like these in future.

cjjdespres avatar Feb 21 '22 14:02 cjjdespres

From job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/265/, failure in SharedClassesAPI_1, with output

JVMT Sun Feb 20 02:26:54 2022
JVMT Incorrect number of caches found. Expected at least 4, found 2
JVMT Sun Feb 20 02:26:54 2022
JVMT Iterating over caches: ERROR
STF 02:26:55.620 - Monitoring Report Summary:
STF 02:26:55.620 -   o Process JVMT ended with the expected exit code (0)
**FAILED** at step 71 (Verify Classes Caches using JVMTI). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_16453495033600/SharedClassesAPI_1/20220220-022207-SharedClassesAPI/execute.pl line 2534.

Version information:

04:19:38  openjdk version "1.8.0_332-internal"
04:19:38  OpenJDK Runtime Environment (build 1.8.0_332-internal-jenkins_2022_02_19_23_57-b00)
04:19:38  Eclipse OpenJ9 VM (build master-ce1c984f065, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20220220_921 (JIT enabled, AOT enabled)
04:19:38  OpenJ9   - ce1c984f065
04:19:38  OMR      - 64690f622a7
04:19:38  JCL      - bd2e6cf7984 based on jdk8u332-b02)

Appears to be tracked by #13239.

cjjdespres avatar Feb 21 '22 14:02 cjjdespres

From build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/274/, failures in

  • cmdLineTester_dumpromclasstests_0
  • cmdLineTester_pltest_j9sig_ext_0
  • testSCCMLTests4_0
  • cmdLineTester_shrcdbgddrext_0
  • testDDRExt_General_0
  • testDDRExt_Callsites_0
  • testDDRExt_JITExt_0
  • testDDRExtJunit_CollisionResilientHashtable_0
  • testDDRExtJunit_MonitorsAndDeadlock1_0
  • testDDRExtJunit_MonitorsAndDeadlock2_0
  • testDDRExtJunit_MonitorsAndDeadlock3_0
  • testDDRExtJunit_MonitorsAndDeadlock3_1
  • testDDRExtJunit_MonitorsAndDeadlock4_0
  • testDDRExtJunit_MonitorsAndDeadlock5_0
  • testDDRExtJunit_MonitorsAndDeadlock5_1
  • testDDRExtJunit_MonitorsAndDeadlock6_0
  • testDDRExtJunit_MonitorsAndDeadlock6_1
  • testRASAPI_0

These have a moderate overlap with the failures in this build. It looks like they all fail because a core dump file is not created, so they all fall under #5310 if I'm right about the connection between that issue and the previous failures. Note that these failures happened on ub20x64rt1-1 again.

cjjdespres avatar Feb 24 '22 14:02 cjjdespres

From job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/271/, failure in cmdLineTester_fieldwatchtests_0, with output:

Testing: fw001-5
Test start time: 2022/02/24 02:15:55 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer  -Xdump  -agentlib:jvmtitest=test:fw001 -XX:+JITInlineWatches -cp "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 6 milliseconds
Time spent executing: 45897 milliseconds
Test result: FAILED
Output from test:

 [OUT] *ERROR*: Invalid number of field reports detected: 0. Expected: 5
 [OUT] 
 [OUT] 
 [OUT] ===============Disabling watches for instanceDoubleField
 [OUT] 
 [OUT] 
 [OUT] *ERROR*: Unable to remove field watches for instanceDoubleField
 [OUT] 
 [OUT] ===============adding field read watches for instanceIntField
 [OUT] 
 [OUT] 
 [OUT] *ERROR*: Unable to add field watches for instanceIntField

 [OUT] =============== Ending - STATIC AND INSTANCE FIELD WATCH TESTS
 [OUT] 
 [OUT] 
 [OUT] *** Test took 45609 milliseconds
 [OUT] FAILED
 [OUT] 
 [OUT] ield Modification Report #5: method: 0x79dd98160de8 location: 22, fieldID: 0x79dd98160e88, signature: F
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceSingleField. The fieldID is: 0x79dd98160e88 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceSingleField, signature: F, isStatic: No, modificationType: remove
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceLongField. The fieldID is: 0x79dd98160eb0 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceLongField, signature: J, isStatic: No, modificationType: add
 [OUT] Field Modification Report #1: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #2: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #3: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #4: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #5: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceLongField. The fieldID is: 0x79dd98160eb0 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceLongField, signature: J, isStatic: No, modificationType: remove
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR] 	at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR] 	at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
>> Success condition was not found: [Return code: 0]

There is more in the log that may be relevant, including more error lines, but I hope that is representative.

Version information:

04:43:16  openjdk version "1.8.0_332-internal"
04:43:16  OpenJDK Runtime Environment (build 1.8.0_332-internal-jenkins_2022_02_24_00_02-b00)
04:43:16  Eclipse OpenJ9 VM (build master-016ce8a1e28, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20220224_925 (JIT enabled, AOT enabled)
04:43:16  OpenJ9   - 016ce8a1e28
04:43:16  OMR      - bb570045546
04:43:16  JCL      - bd2e6cf7984 based on jdk8u332-b02)

Reproduced with a 1/50 failure rate with JITAAS in view/Test_grinder/job/Grinder/21338/, and not reproduced with a 100x job in view/Test_grinder/job/Grinder/21375/ without JITAAS.

cjjdespres avatar Feb 24 '22 22:02 cjjdespres

Very large number of test failures over the past few days. It will take a little while to sort through it all.

cjjdespres avatar Feb 28 '22 15:02 cjjdespres

I've noticed that there's a large overlap between the failing tests and the ones in this comment, but that may just be coincidence?

Edit: Even though there is such a large overlap (including tests that I haven't had a chance to look at yet), the ones in that comment all failed due to the disabling of SecurityManager, and so it's not directly related.

cjjdespres avatar Feb 28 '22 16:02 cjjdespres

Two jobs had tests that all failed in the same way - the processes could not connect to a VM, as Monitored VM not ready, then eventually gave up and failed. These failures occurred in:

job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/276/

  • TestJlmRemoteClassAuth_1
  • TestJlmRemoteThreadAuth_1
  • TestIBMJlmRemoteMemoryAuth_SE80_Linux_0
  • TestIBMJlmRemoteMemoryNoAuth_SE80_Linux_0
  • TestIBMJlmRemoteMemoryNoAuth_SE80_Linux_1

job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/275/

  • TestJlmRemoteClassAuth_0
  • TestJlmRemoteClassNoAuth_1
  • TestJlmRemoteMemoryNoAuth_0
  • TestJlmRemoteNotifierProxyAuth_1
  • TestJlmRemoteThreadAuth_0
  • TestJlmRemoteThreadNoAuth_0
  • TestIBMJlmRemoteClassAuth_SE80_Linux_1
  • TestIBMJlmRemoteClassNoAuth_SE80_Linux_1

Representative log output:

STF 04:44:26.517 - +------ Step 5 - Wait for the processes to complete
STF 04:44:26.517 - | Wait for processes to meet expectations
STF 04:44:26.517 - |   Processes: [LT1, CL1]
STF 04:44:26.517 - |
STF 04:44:26.517 - Monitoring processes: CL1 LT1
CL1 j> 2022/02/27 04:44:26.692 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/02/27 04:44:26.728 Attempting to connect
CL1 j> 2022/02/27 04:44:26.910 Monitored VM not ready at Feb 27, 2022 4:44:26 AM (attempt 1, elapsed 163ms).
CL1 j> 2022/02/27 04:44:26.910 Waiting 5 secs and trying again...
CL1 j> 2022/02/27 04:44:31.912 Attempting to connect
CL1 j> 2022/02/27 04:44:33.498 Monitored VM not ready at Feb 27, 2022 4:44:33 AM (attempt 2, elapsed 6769ms).
CL1 j> 2022/02/27 04:44:33.498 Waiting 5 secs and trying again...
CL1 j> 2022/02/27 04:44:38.499 Attempting to connect
CL1 j> 2022/02/27 04:44:38.587 Monitored VM not ready at Feb 27, 2022 4:44:38 AM (attempt 3, elapsed 11858ms).

CL1 j> 2022/02/27 04:46:55.214 Failed to connect to Monitored VM after 30 attempts in 148 seconds - giving up.  Connection Exception received is below:
CL1 stderr java.rmi.ConnectException: Connection refused to host: 9.24.248.182; nested exception is: 
CL1 stderr 	java.net.ConnectException: Connection refused (Connection refused)
CL1 stderr 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
CL1 stderr 	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
CL1 stderr 	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
CL1 stderr 	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
CL1 stderr 	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
CL1 stderr 	at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
CL1 stderr 	at com.sun.proxy.$Proxy0.newClient(Unknown Source)
CL1 stderr 	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
CL1 stderr 	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
CL1 stderr 	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ServerConnector.doConnect(ServerConnector.java:273)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ServerConnector.getSecureServerConnection(ServerConnector.java:170)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ServerConnector.<init>(ServerConnector.java:82)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.<init>(ClassProfiler.java:72)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.main(ClassProfiler.java:89)
CL1 stderr Caused by: java.net.ConnectException: Connection refused (Connection refused)
CL1 stderr 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
CL1 stderr 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
CL1 stderr 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
CL1 stderr 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
CL1 stderr 	at java.net.Socket.connect(Socket.java:607)
CL1 stderr 	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288)
CL1 stderr 	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:147)
CL1 stderr 	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
CL1 stderr 	at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:121)
CL1 stderr 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
CL1 stderr 	... 14 more
CL1 stderr Exception in thread "main" java.lang.NullPointerException
CL1 stderr 	at com.ibm.java.lang.management.internal.ManagementUtils.isANotificationEmitter(ManagementUtils.java:536)
CL1 stderr 	at java.lang.management.ManagementFactory.newPlatformMXBeanProxy(ManagementFactory.java:289)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.getStatsViaProxy(ClassProfiler.java:117)
CL1 stderr 	at net.adoptopenjdk.test.jlm.remote.ClassProfiler.main(ClassProfiler.java:102)
STF 04:46:55.496 - **FAILED** Process CL1 ended with exit code (1) and not the expected exit code/s (0)
STF 04:46:55.496 - Monitoring Report Summary:
STF 04:46:55.496 -   o Process CL1 ended with exit code (1) and not the expected exit code/s (0)
STF 04:46:55.496 -   o Process LT1 is still running as expected
STF 04:46:55.496 - Killing processes: CL1 LT1
STF 04:46:55.496 -   o Process LT1 pid 2478 stop()
STF 04:47:05.549 -   o Process LT1 pid 2478 terminate()
STF 04:47:06.550 -   o Process LT1 pid 2478 killed
STF 04:47:06.550 -   o Process CL1 pid 2479 is not running
**FAILED** at step 5 (Wait for the processes to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_16459519958390/TestJlmRemoteClassAuth_1/20220227-044425-TestJlmRemoteClassAuth/execute.pl line 178.
STF 04:47:06.792 - **FAILED** execute script failed. Expected return value=0 Actual=1

Haven't attempted to reproduce.

cjjdespres avatar Feb 28 '22 16:02 cjjdespres

From job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/191/, failures in

  • TestJlmRemoteMemoryNoAuth_0
  • TestJlmRemoteNotifierProxyAuth_0
  • TestJlmRemoteThreadNoAuth_0
  • TestJlmRemoteThreadNoAuth_1
  • TestIBMJlmRemoteClassAuth_0
  • TestIBMJlmRemoteClassAuth_1
  • TestJlmRemoteClassAuth_1

These all fail in similar ways - the tests attempt to connect to a particular ServerURL, and either just before or just after that a child process fails unexpectedly and the test is aborted.

Representative console log:

STF 02:00:23.344 - +------ Step 3 - Wait for processes to complete
STF 02:00:23.344 - | Wait for processes to meet expectations
STF 02:00:23.344 - |   Processes: [LT1, CL1]
STF 02:00:23.344 - |
STF 02:00:23.345 - Monitoring processes: CL1 LT1
CL1 j> 2022/02/26 02:00:25.833 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/02/26 02:00:26.710 Attempting to connect
STF 02:00:28.587 - **FAILED** Process LT1 has ended unexpectedly
STF 02:00:28.587 - Monitoring Report Summary:
STF 02:00:28.587 -   o Process CL1 is still running
STF 02:00:28.587 -   o Process LT1 has ended unexpectedly
STF 02:00:28.588 - Killing processes: CL1 LT1
STF 02:00:28.588 -   o Process LT1 pid 26108 is not running
STF 02:00:28.588 -   o Process CL1 pid 26109 stop()
STF 02:00:38.682 -   o Process CL1 pid 26109 terminate()
STF 02:00:39.700 -   o Process CL1 pid 26109 killed
CL1 j> 2022/02/26 02:00:29.574 Connection established!
CL1 j> 2022/02/26 02:00:30.503 Starting to write data
**FAILED** at step 3 (Wait for processes to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_1645867773491/TestJlmRemoteMemoryNoAuth_0/20220226-020012-TestJlmRemoteMemoryNoAuth/execute.pl line 163.
STF 02:00:39.773 - **FAILED** execute script failed. Expected return value=0 Actual=1

Haven't attempted to reproduce.

cjjdespres avatar Feb 28 '22 16:02 cjjdespres

I've noticed that there's a large overlap between the failing tests and the ones in https://github.com/eclipse-openj9/openj9/issues/14412#issuecomment-1027354827, but that may just be coincidence?

If running Java 18+ you need the following set -Djava.security.manager=allow. We do that for the regular tests by setting the JAVA_TOOL_OPTIONS environment variable to prepend the option https://github.com/adoptium/aqa-tests/pull/3352

pshipton avatar Feb 28 '22 17:02 pshipton

The next jobs all fail due to a segfault. They also have some tests that seem to fail for other reasons, so I'll post those separately. The two running the functional tests all have identical stack backtraces

----------- Stack Backtrace -----------
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x133da (0x00007F0F643EF1BA [libj9vm29.so+0xa61ba])
bytecodeLoopCompressed+0x95 (0x00007F0F643DBDD5 [libj9vm29.so+0x92dd5])
 (0x00007F0F6447F5B2 [libj9vm29.so+0x1365b2])

aside from what I assume are the memory addresses. The one running the system tests has more varied stack backtraces. The jobs and tests are:

job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/

  • SyntheticGCWorkload_DoubleMap_J9_0
  • testSCCacheManagement_0
  • cmdLineTester_classesdbgddrext_1
  • cmdLineTester_defaultLazySymbolResolution_1
  • cmdLineTester_fastClassHashTable_0
  • cmdLineTester_hangTest_0
  • cmdLineTester_javaAssertions_0
  • cmdLineTester_decompilationTests_nongold_3
  • cmdLineTester_lockWordAlignment_Object_Standard_0
  • cmdLineTester_lockWordAlignment_Object_iii_0
  • cmdLineTester_lockWordAlignment_Object_d_0
  • cmdLineTester_modularityddrtests17_1
  • cmdLineTester_shareClassesBadStackMap_0
  • testSCCMLTests3_1
  • testSCCMLTests4_1
  • testSCCMLTests6_1
  • testSCCMLSnapshot_1
  • testSCCMLAotMethodOperation_1
  • cmdLineTester_shrcdbgddrext_1
  • cmdLineTest_stackSizeInfoTest_0
  • cmdLineTester_verbosetest_5
  • cmdLineTester_vmRuntimeState_0
  • cmdLineTester_xlogTests_0
  • StackWalkerTestJava10_0
  • JCL_TEST_Java-Lang-Invoke_0
  • TestAttachAPIEnabling_0
  • TestSunAttachClasses_0
  • gcNotificationTest_Optthruput_1
  • gcNotificationTest_OptAvgpause_1
  • gcNotificationTest_Balanced_0
  • gcNotificationTest_Balanced_1
  • gcNotificationTest_Metronome_1
  • ContendedFieldsTests_90_0
  • threadMXBeanTestSuite2_2
  • threadMXBeanTimedParkTest_2
  • threadMXBeanTimedParkTest_3
  • memoryCategories_0
  • HashConsistency_0
  • jniOnLoadExceptions_1
  • badUTF8inJNI_1
  • badUTF8inJNI_2
  • testvmcheck_3
  • testvmcheck_6
  • TestSoftMxCallBackOOM_1
  • TestSoftMxCallBackExtend_1
  • MethodVisibilityTests_0
  • UnreflectTests_0

job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/

  • SyntheticGCWorkload_DoubleMap_J9_0
  • SecurityTests_0
  • CDSAdaptorTest_0
  • cmdLineTester_bootStrapStaticVerify_0
  • cmdLineTester_EnableAssertionStatusTest_0
  • cmdLineTest_J9test_extended_0
  • cmdLineTest_J9test_consoleUpTo17_0
  • cmdLineTester_defaultLazySymbolResolution_1
  • cmdLineTester_dumpromclasstests_0
  • cmdLineTester_getCallerClassTests_SE80_0
  • cmdLineTester_hangTest_0
  • cmdLineTester_ignoreunrecognizedvmoptions_0
  • cmdLineTester_jep178_staticLinking_SE80_0
  • cmdLineTester_fieldwatchtests_0

job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/

  • DaaLoadTest_all_5m_1
  • MiniMix_5m_0
  • DBBLoadTest_5m_0
  • HCRLateAttachWorkload_previewEnabled_0
  • ObjectTreeLoadTest_5m_1
  • SharedClasses.SCM01.SingleCL_1
  • SharedClasses.SCM01.MultiCL_1
  • SharedClasses.SCM01.MultiThread_1
  • SharedClasses.SCM23.MultiThreadMultiCL_1

Haven't been able to reproduce.

cjjdespres avatar Feb 28 '22 17:02 cjjdespres

From job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/ (also discussed above), failures in

  • SecurityTests_0
  • cmdLineTester_dumpromclasstests_0
  • cmdLineTester_locales_0
  • cmdLineTester_lockWordAlignment_Object_i_0
  • testDDRExtJunit_MonitorsAndDeadlock2_1
  • testDDRExtJunit_MonitorsAndDeadlock3_1
  • testDDRExtJunit_MonitorsAndDeadlock6_1
  • TestAttachAPI_0
  • TestAttachErrorHandling_0
  • gcNotificationTest_GenCon_0
  • threadMXBeanTimersTest_3
  • HashPerformance_0
  • ContendedFieldsTests_90_1
  • ContendedFieldsTests_90_3

They all seem to fail in the same way, but some have less detail in their errors. All but the ContendedFieldsTests and the testDDRExtJunit tests fail with messages like:

TestAttachErrorHandling_0 Start Time: Mon Feb 28 00:49:39 2022 Epoch Time (ms): 1646038179205
variation: NoOptions
JVM_OPTIONS: -XX:+UseJITServer 
Failed to register natives for jdk.internal.reflect.ConstantPool
Exception in thread "main" java/lang/ExceptionInInitializerError
	at java/lang/J9VMInternals.ensureError ([email protected]/J9VMInternals.java:185)
	at java/lang/J9VMInternals.recordInitializationFailure ([email protected]/J9VMInternals.java:174)
java/lang/NullPointerException
	at com/ibm/jit/JITHelpers.getClassDepthAndFlagsFromJ9Class64 ([email protected]/NativeMethod:4294967295)
	at com/ibm/jit/JITHelpers.isArray ([email protected]/JITHelpers.java:141)
	at java/lang/J9VMInternals.fastIdentityHashCode ([email protected]/J9VMInternals.java:408)
	at java/lang/Object.hashCode ([email protected]/Object.java:120)
	at java/util/HashMap.hash ([email protected]/HashMap.java:338)
	at java/util/HashMap.getNode ([email protected]/HashMap.java:568)
	at java/util/HashMap.get ([email protected]/HashMap.java:556)
	at jdk/internal/reflect/Reflection.registerFilter ([email protected]/Reflection.java:286)
	at jdk/internal/reflect/Reflection.registerFieldsToFilter ([email protected]/Reflection.java:273)
	at jdk/internal/reflect/ConstantPool.<clinit> ([email protected]/ConstantPool.java:108)

TestAttachErrorHandling_0_FAILED

The testDDREExtJunit tests have their TAP results cut off, but in the console log they have that same error but without the "failed to register natives" line. The ContendedFieldTests have only a NullPointerException reported.

Haven't been able to reproduce.

cjjdespres avatar Feb 28 '22 18:02 cjjdespres

From job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, also discussed above, failure in

  • StackWalkerTest_1

Console log:

Exception in thread "main" Exception in thread "Attach API initializer" java.lang.InternalError: not invoker: invokeExactjava.lang.InternalError: not invoker: invokeExact

	at java.base/java.lang.invoke.Invokers.methodHandleInvokeLinkerMethod(Invokers.java:260)	at java.base/java.lang.invoke.Invokers.methodHandleInvokeLinkerMethod(Invokers.java:260)

	at java.base/java.lang.invoke.MethodHandleNatives.linkMethodImpl(MethodHandleNatives.java:482)
	at java.base/java.lang.invoke.MethodHandleNatives.linkMethod(MethodHandleNatives.java:473)
	at java.base/java.lang.invoke.MethodHandleNatives.linkMethodImpl(MethodHandleNatives.java:482)	at java.base/java.lang.invoke.MethodHandleResolver.linkCallerMethod(MethodHandleResolver.java:457)

	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
	at java.base/java.lang.invoke.MethodHandleNatives.linkMethod(MethodHandleNatives.java:473)	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:196)

	at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.open(SystemModuleFinders.java:467)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findResourceAsStream(BuiltinClassLoader.java:488)
	at java.base/jdk.internal.loader.BootLoader.findResourceAsStream(BootLoader.java:174)
	at java.base/java.lang.invoke.MethodHandleResolver.linkCallerMethod(MethodHandleResolver.java:457)	at java.base/java.lang.Module.getResourceAsStream(Module.java:1671)

	at java.base/com.ibm.oti.vm.MsgHelp.loadMessages(MsgHelp.java:185)
	at java.base/com.ibm.oti.util.PriviAction.run(PriviAction.java:172)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:683)
	at java.base/com.ibm.oti.util.Msg.<clinit>(Msg.java:63)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:874)
	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:868)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)	at java.base/java.lang.Thread.uncaughtException(Thread.java:1403)

	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:196)
	at java.base/java.util.ResourceBundle.getLoader(ResourceBundle.java:584)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1574)
	at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1549)
	at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:858)
	at java.base/sun.launcher.LauncherHelper$ResourceBundleHolder.<clinit>(LauncherHelper.java:125)
	at java.base/sun.launcher.LauncherHelper.getLocalizedMessage(LauncherHelper.java:483)
	at java.base/sun.launcher.LauncherHelper.abort(LauncherHelper.java:647)
	at java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:807)
	at java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:686)

StackWalkerTest_1_FAILED

Haven't been able to reproduce.

cjjdespres avatar Feb 28 '22 18:02 cjjdespres

From job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, also discussed above, failure in:

  • MathLoadTest_all_5m_0

Excerpt from console log:

MLT stderr Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.logging.log4j.message.DefaultFlowMessageFactory
MLT stderr 	at org.apache.logging.log4j.spi.AbstractLogger.<clinit>(AbstractLogger.java:101)
MLT stderr 	at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:61)
MLT stderr 	at net.adoptopenjdk.loadTest.LoadTest.<clinit>(LoadTest.java:46)
MLT stderr Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.DefaultFlowMessageFactory
MLT stderr 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:827)
MLT stderr 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
MLT stderr 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1091)
MLT stderr 	... 3 more
STF 01:41:55.438 - **FAILED** Process MLT ended with exit code (1) and not the expected exit code/s (0)

This seems more like a misconfigured system or a failure in the test itself, since the test can't find some component of log4j.

Haven't been able to reproduce.

cjjdespres avatar Feb 28 '22 18:02 cjjdespres

From job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/, also discussed above, failure in:

  • cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2

Excerpt from console log:

===============================================
Running test cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2 ...
===============================================
cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2 Start Time: Mon Feb 28 01:30:51 2022 Epoch Time (ms): 1646040651529
variation: Mode607-OSRG
JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation 
Error: Could not find or load main class MainTester

cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2_FAILED

This also seems like a failure in the test itself.

Haven't been able to reproduce.

cjjdespres avatar Feb 28 '22 18:02 cjjdespres

From job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/274/, failure in:

  • MiniMix_aot_5m_0

Console log excerpt:

04:40:57  STF 01:40:57.545 - Monitoring processes: LT 
04:40:58  LT  stderr Exception in thread "main" java/lang/NoClassDefFoundError: java.ut����.����������������.����������������
04:40:58  LT  stderr 	at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString ([email protected]/MethodTypeHelper.java:259)
04:40:58  LT  stderr 	at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic ([email protected]/MethodHandleResolver.java:203)
04:40:58  LT  stderr 	at jdk/internal/module/ModuleBootstrap.decode ([email protected]/ModuleBootstrap.java:882)
04:40:58  LT  stderr 	at jdk/internal/module/ModuleBootstrap.decode ([email protected]/ModuleBootstrap.java:905)
04:40:58  LT  stderr 	at jdk/internal/module/ModuleBootstrap.addExtraExportsAndOpens ([email protected]/ModuleBootstrap.java:674)
04:40:58  LT  stderr 	at jdk/internal/module/ModuleBootstrap.boot ([email protected]/ModuleBootstrap.java:432)
04:40:58  LT  stderr 	at java/lang/ClassLoader.initializeClassLoaders ([email protected]/ClassLoader.java:211)
04:40:58  LT  stderr 	at java/lang/Thread.initialize ([email protected]/Thread.java:462)
04:40:58  LT  stderr 	at java/lang/Thread.<init> ([email protected]/Thread.java:162)
04:40:58  STF 01:40:57.816 - **FAILED** Process LT  ended with exit code (1) and not the expected exit code/s (0)
04:40:58  STF 01:40:57.817 - Monitoring Report Summary:
04:40:58  STF 01:40:57.817 -   o Process LT  ended with exit code (1) and not the expected exit code/s (0)
04:40:58  STF 01:40:57.817 - Killing processes: LT 
04:40:58  STF 01:40:57.817 -   o Process LT  pid 2374985 is not running
04:40:58  **FAILED** at step 1 (Run mixed unit tests). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_16460406309573/MiniMix_aot_5m_0/20220228-014051-MixedLoadTest/execute.pl line 98.
04:40:58  STF 01:40:57.826 - **FAILED** execute script failed. Expected return value=0 Actual=1

The TAP extended test results are cut off partway through the first line of the error; this is from the full console log.

cjjdespres avatar Feb 28 '22 20:02 cjjdespres