aqa-tests
aqa-tests copied to clipboard
adoptium
Test fails but jenkins status is green, TAP test results shows "No tests"
Describe the bug Test fails but jenkins status is green. The jenkins parent job did see the failure, otherwise it would never have been noticed. However I wonder if there are tests failing and going unnoticed when there is no jenkins parent job.
See https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_sanity.functional_x86-64_windows_Release_testList_1/26
ShareClassesCMLOpenJ9_1_FAILED, however the TAP test results shows "No tests" and the run is green.
To Reproduce Not sure if it's easily reproduced.
Expected behavior An amber test result from jenkins.
TKG summary and its generated .tap file are correct.
13:46:00 TEST TARGETS SUMMARY
13:46:00 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
13:46:00 DISABLED test targets:
13:46:00 cmdLineTester_pltest_0
13:46:00 cmdLineTester_pltest_tty_extended_0
13:46:00 cmdLineTester_pltest_numcpus_notBound_0
13:46:00 jsigjnitest_1
13:46:00 recreateClassTest_0
13:46:00 CFdumptest_0
13:46:00
13:46:00 PASSED test targets:
13:46:00 cmdLineTester_test_0
13:46:00 cmdLineTest_J9test_common_0
13:46:00 cmdLineTester_GCCheck_0
13:46:00 cmdLineTester_GCCheck_2
13:46:00 cmdLineTester_GCCheck_3
13:46:00 cmdLineTester_GCRegressionTests_3
13:46:00 cmdLineTester_GCRotatingVerboseLogTests_2
13:46:00 cmdLineTester_GCRotatingVerboseLogTests_4
13:46:00 cmdLineTester_gcsuballoctests_0
13:46:00 cmdLineTester_jvmtitests_0
13:46:00 cmdLineTester_jvmtitests_1
13:46:00 cmdLineTester_jvmtitests_2
13:46:00 cmdLineTester_jvmtitests_8
13:46:00 cmdLineTester_jvmtitests_hcr_SE80_0
13:46:00 cmdLineTester_jvmtitests_hcr_SE80_2
13:46:00 cmdLineTester_jvmtitests_hcr_SE80_8
13:46:00 cmdLineTester_jvmtitests_hcr_SE80_9
13:46:00 cmdLineTester_jvmtitests_debug_1
13:46:00 cmdLineTester_jvmtitests_debug_3
13:46:00 cmdLineTester_jvmtitests_debug_6
13:46:00 cmdLineTester_jvmtitests_debug_11
13:46:00 cmdLineTester_jvmtitests_debug_openj9_none_SCC_1
13:46:00 cmdLineTester_jvmtitests_debug_openj9_none_SCC_2
13:46:00 cmdLineTester_jvmtitests_debug_openj9_none_SCC_6
13:46:00 cmdLineTester_jvmtitests_debug_openj9_none_SCC_9
13:46:00 cmdLineTester_jvmtitests_debug_openj9_none_SCC_11
13:46:00 cmdLineTester_decompilationTests_0
13:46:00 cmdLineTester_jython_0
13:46:00 cmdLineTester_jython_1
13:46:00 cmdLineTester_jython_3
13:46:00 cmdLineTester_jython_6
13:46:00 cmdLineTester_libpathTestRtfChild_0
13:46:00 cmdLineTester_loadLibraryTests_0
13:46:00 cmdLineTester_CryptoTest_0
13:46:00 cmdLineTester_DataHelperTests_1
13:46:00 testSCCMLTests1_openj9_0
13:46:00 cmdLineTester_SCHelperCompatTests_win_1
13:46:00 cmdLineTester_ShareClassesSimpleSanity_0
13:46:00 cmdLineTester_SCURLHelperTests_80_0
13:46:00 cmdLineTester_SCURLHelperTests_80_1
13:46:00 SCURLClassLoaderTests_SE80_0
13:46:00 SCURLClassLoaderNPTests_SE80_1
13:46:00 cmdLineTester_XcheckJNI_0
13:46:00 testXXArgumentTesting_j9_1
13:46:00 testXXArgumentTesting_j9_3
13:46:00 testXXArgumentTesting_j9_4
13:46:00 regressionFastresolve_mode110_0
13:46:00 regressionFastresolve_mode150_0
13:46:00 jsr335_interfacePrivateMethod_1
13:46:00 CtorBCVTests_0
13:46:00 JCL_Test_SE80_1
13:46:00 JCL_Test_SE80_Native_0
13:46:00 JCL_Test_OutOfMemoryError_SE80_0
13:46:00 J9VMInternals_Test_SE80_0
13:46:00 JCL_TEST_Java-Security_SE80_0
13:46:00 JCL_Test_JITHelpers_SE80_1
13:46:00 JCL_Test_JITHelpers_SE80_3
13:46:00 TestArrayCopy_0
13:46:00 TestArrayCopy_2
13:46:00 TestArrayCopy_openj9_none_SCC_0
13:46:00 TestArrayCopy_openj9_none_SCC_2
13:46:00 pthreadDestructor_0
13:46:00 testReflectInvoke_0
13:46:00 NoSuchMethodTests_0
13:46:00 hanoiTest_0
13:46:00 memleaksTest_0
13:46:00 testStringInterning_1
13:46:00 testStringStreams_0
13:46:00 floatSanityTests_1
13:46:00 classRelationshipVerifierTests_0
13:46:00 DupClassNameTest_0
13:46:00 RuntimeAnnotationTests_0
13:46:00 TestFlushReflectionCache_SE80_0
13:46:00 TestRefreshGCSpecialClassesCache_NOBCI_SE80_0
13:46:00 TestRefreshGCSpecialClassesCache_NOBCI_SE80_1
13:46:00 TestRefreshGCSpecialClassesCache_BCI_EXTENDED_HCR_SE80_0
13:46:00 stringConcatOptTest_0
13:46:00 stringConcatOptTest_1
13:46:00 jit_jitt_1
13:46:00 jit_jitt_array_0
13:46:00 jit_jitt_array_1
13:46:00 jit_jitt_array_3
13:46:00 jit_jitt_array_5
13:46:00 jit_jitt_array_compress_0
13:46:00 jit_jitt_array_compress_2
13:46:00 jit_compareAndBranch_0
13:46:00 jit_jitt_openj9_none_SCC_1
13:46:00 jit_jitt_openj9_none_SCC_3
13:46:00 jit_tr_0
13:46:00 SIMDCommonedAddressTest_0
13:46:00 SeqLoadSimplificationTest_0
13:46:00 jit_ra_0
13:46:00 jit_recognizedMethod_0
13:46:00 jit_recognizedMethod_2
13:46:00 jit_recognizedMethod_4
13:46:00 jit_recognizedMethod_6
13:46:00 JLM_Tests_interface_SE80_1
13:46:00 JLM_Tests_class_SE80_1
13:46:00 JLM_Tests_IBMinternal_SE80_1
13:46:00 jsr335tests_SE80_1
13:46:00 jsr335tests_SE80_3
13:46:00 jsr335tests_SE80_5
13:46:00 jsr335tests_defendersupersends_SE80_0
13:46:00 algotest2_0
13:46:00 jniargtests_0
13:46:00 jniargtests_2
13:46:00 gc_rwlocktest_win_0
13:46:00 shrtest_win_SE80_1
13:46:00 thrstatetest_1
13:46:00 SharedCPEntryInvokerTests_1
13:46:00 UnsafeTests_SE80_0
13:46:00 UnsafeTests_SE80_2
13:46:00 J9vmTest_3
13:46:00 J9vmTest_5
13:46:00
13:46:00 FAILED test targets:
13:46:00 ShareClassesCMLOpenJ9_1
13:46:00
13:46:00 TOTAL: 149 EXECUTED: 115 PASSED: 114 FAILED: 1 DISABLED: 6 SKIPPED: 28
.tap file: https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_sanity.functional_x86-64_windows_Release_testList_1/26/artifact/aqa-tests/TKG/output_16579891828052/Test_openjdk8_j9_sanity.functional_x86-64_windows_Release_testList_1.tap/view/
I suspect this issue is related to .tap file not getting processed correctly via the Jenkins TAP plugin.
Another failure example is https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.functional_x86-64_windows_Release_testList_1/26
Another failure example https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_x86-64_windows_Release_testList_4/25
In that last failure (https://github.com/adoptium/aqa-tests/issues/3864#issuecomment-1187565958), there should be diagnostic files archived in the artifacts, but there are not. I suspect this is related.
re https://github.com/adoptium/aqa-tests/issues/3864#issuecomment-1187573855, yes, it is related. By default, the test pipeline only archives the test output for the failed build.
Expected Jenkins output for processing TAP file:
09:13:18 TAP Reports Processing: START
09:13:18 Looking for TAP results report in workspace using pattern: **/*.tap
09:13:18 Saving reports...
09:13:19 Processing '/var/lib/jenkins/jobs/Test_openjdk11_j9_special.system_x86-64_windows_Release_testList_4/builds/24/tap-master-files/aqa-tests/TKG/output_16574410139152/Test_openjdk11_j9_special.system_x86-64_windows_Release_testList_4.tap'
09:13:19 Parsing TAP test result [/var/lib/jenkins/jobs/Test_openjdk11_j9_special.system_x86-64_windows_Release_testList_4/builds/24/tap-master-files/aqa-tests/TKG/output_16574410139152/Test_openjdk11_j9_special.system_x86-64_windows_Release_testList_4.tap].
09:13:19 TAP Reports Processing: FINISH
Actual output (Found matching files but did not find any TAP results):
12:25:25 TAP Reports Processing: START
12:25:25 Looking for TAP results report in workspace using pattern: **/*.tap
12:25:25 Saving reports...
12:25:26 Found matching files but did not find any TAP results.
[Pipeline] junit
12:25:26 Recording test results
12:25:27 None of the test reports contained any result
12:25:27 [Checks API] No suitable checks publisher found.
Another one https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/362
00:39:36 FAILED test targets:
00:39:36 testDDRExt_Class_0
00:39:36
00:39:36 TOTAL: 77 EXECUTED: 77 PASSED: 76 FAILED: 1 DISABLED: 0 SKIPPED: 0
00:39:36 cat: 'C:/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/aqa-tests/TKGoutput_1663385924712../AQACert.log': No such file or directory
00:39:36 cat: 'C:/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/aqa-tests/TKGoutput_1663385924712../SHA.txt': No such file or directory
00:39:36 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
00:39:36
00:39:36 _testList done
00:39:36 make[2]: Leaving directory '/cygdrive/c/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/aqa-tests/TKG'
00:39:36 make[1]: Leaving directory '/cygdrive/c/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/aqa-tests/TKG'
[Pipeline] sh
00:39:37 ++ uname
00:39:37 + '[' CYGWIN_NT-10.0-17763 = AIX ']'
00:39:37 ++ uname
00:39:37 + '[' CYGWIN_NT-10.0-17763 = SunOS ']'
00:39:37 ++ uname
00:39:37 + '[' CYGWIN_NT-10.0-17763 = '*BSD' ']'
00:39:37 + MAKE=make
00:39:37 + make -f ./aqa-tests/TKG/testEnv.mk testEnvTeardown
00:39:37 make: Nothing to be done for 'testEnvTeardown'.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Post)
[Pipeline] step
00:39:37 TAP Reports Processing: START
00:39:37 Looking for TAP results report in workspace using pattern: **/*.tap
00:39:37 Saving reports...
00:39:38 Found matching files but did not find any TAP results.
[Pipeline] junit
00:39:38 Recording test results
00:39:38 None of the test reports contained any result
00:39:38 [Checks API] No suitable checks publisher found.
[Pipeline] echo
00:39:38 Saving aqa-tests/testenv/testenv.properties file on jenkins.
[Pipeline] archiveArtifacts
00:39:38 Archiving artifacts
00:39:38 Recording fingerprints
00:39:38 No test report files were found. Configuration error?
In the code, we specified TAP failIfNoResults: true at both the child and parent build level.
https://github.com/adoptium/aqa-tests/blob/9ca37a6a54e8e4fabfd785b80bc6936cb4bf3ec5/buildenv/jenkins/JenkinsfileBase#L705
https://github.com/adoptium/aqa-tests/blob/9ca37a6a54e8e4fabfd785b80bc6936cb4bf3ec5/buildenv/jenkins/JenkinsfileBase#L1082
In this case, it worked at the parent level, but not the child level: parent:
04:24:04.737 TAP Reports Processing: START
04:24:04.737 Looking for TAP results report in workspace using pattern: Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/362/**/*.tap
04:24:04.845 Saving reports...
04:24:05.260 Processing '/var/lib/jenkins/jobs/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly/builds/370/tap-master-files/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/362/aqa-tests/TKG/output_1663385924712/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0.tap'
04:24:05.260 Parsing TAP test result [/var/lib/jenkins/jobs/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly/builds/370/tap-master-files/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0/362/aqa-tests/TKG/output_1663385924712/Test_openjdk8_j9_extended.functional_x86-32_windows_Nightly_testList_0.tap].
04:24:05.297 There are failed test cases. Marking build as UNSTABLE
04:24:05.349 TAP Reports Processing: FINISH
child:
03:07:31.801 TAP Reports Processing: START
03:07:31.802 Looking for TAP results report in workspace using pattern: **/*.tap
03:07:32.301 Saving reports...
03:07:32.463 Found matching files but did not find any TAP results.
...
Finished: SUCCESS
I still suspect .tap file did not get processed correctly via the Jenkins TAP plugin. The file was found, but the failed test output may mess up the TAP plugin paring, which resulting did not find any TAP results.
A side affect seems to be that diagnostic artifacts are not collected because a failure was not recognized.
