TEST TARGETS SUMMARY does not capture failed test and resulted false positive test build

[llxia]

runtime-api-java_security_0 failed but is not in TEST TARGETS SUMMARY. The test build finished in SUCCESS, which is a false positive.

related: internal issue

08:09:01  Test results: passed: 448; failed: 1
...
08:09:02  runtime-api-java_security_0_FAILED

10:08:39  TEST TARGETS SUMMARY
10:08:39  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39  DISABLED test targets:
10:08:39  	compiler-api-javax_tools_0
10:08:39  	runtime-api-java_util-regex_0
10:08:39  
10:08:39  PASSED test targets:
10:08:39  	compiler-api-signaturetest_0 - Test results: passed: 2 
10:08:39  	runtime-api-java_lang-instrument_0 - Test results: passed: 10 
10:08:39  	runtime-api-java_lang-invoke_0 - Test results: passed: 110 
10:08:39  	runtime-api-java_lang-reflect_0 - Test results: passed: 143 
10:08:39  	runtime-api-java_lang-Package_0 - Test results: passed: 12 
10:08:39  	runtime-api-java_lang-String_0 - Test results: passed: 64 
10:08:39  	runtime-api-java_lang-StringBuilder_0 - Test results: passed: 25 
10:08:39  	runtime-api-java_lang-StringBuffer_0 - Test results: passed: 28 
10:08:39  	runtime-api-java_lang-ClassLoader_0 - Test results: passed: 35 
10:08:39  	runtime-api-java_lang-ModuleLayer_0 - Test results: passed: 23 
10:08:39  	runtime-api-java_lang-module_0 - Test results: passed: 163 
10:08:39  	runtime-api-java_lang-StackWalker_0 - Test results: passed: 8 
10:08:39  	runtime-api-java_lang-Module_class_0 - Test results: passed: 42 
10:08:39  	runtime-api-java_lang-SecurityManager_0 - Test results: passed: 2 
10:08:39  	runtime-api-java_lang-System_0 - Test results: passed: 26 
10:08:39  	runtime-api-java_lang-Class_0 - Test results: passed: 83 
10:08:39  	runtime-api-signaturetest_0 - Test results: passed: 20 
10:08:39  	runtime-api-modulegraph_0 - Test results: passed: 1 
10:08:39  	runtime-api-javax_naming-spi_0 - Test results: passed: 129 
10:08:39  
10:08:39  TOTAL: 21   EXECUTED: 19   PASSED: 19   FAILED: 0   DISABLED: 2   SKIPPED: 0
10:08:39  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39  
10:08:39  TESTCASES RESULTS SUMMARY: passed: 32,585; failed: 1; error: 0; skipped: 0
10:08:39  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39  ALL TESTS PASSED
...
Finished: SUCCESS

[sophia-guo]

Is this related or similar https://github.com/adoptium/aqa-tests/issues/4092?

[llxia]

The above build has ITERATIONS=1. I do not think it is related to https://github.com/adoptium/aqa-tests/issues/4092

[smlambert]

We have also just recently seen a case of this.

Preliminary questions, has the Jenkins plugin been updated on servers recently and if so, what changes did it have, has there been a regression in it?

[llxia]

Noticed another case on zlinux (link)

07:22:20.715  TEST TARGETS SUMMARY
07:22:20.715  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
07:22:20.715  PASSED test targets:
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  	MachineInfo_0
07:22:20.715  
07:22:20.715  TOTAL: 9   EXECUTED: 9   PASSED: 9   FAILED: 0   DISABLED: 0   SKIPPED: 0
07:22:20.715  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Only MachineInfo_0 in the summary. None of the other tests are tracked in TEST TARGETS SUMMARY.

07:22:20.001  -----------------------------------
07:22:20.001  ParallelStreamsLoadTest_special_J9_27_PASSED
07:22:20.001  -----------------------------------

[llxia]

Another case. In the console output, we ran TESTLIST=openJcePlusTests_0,cmdLineTester_vmRuntimeState_0,testDDRExt_Callsites_0,testSCCMLTests6_0,testSCCMLTests5_1,cmdLineTester_jvmtitests_extended_nongold_6,cmdLineTester_jvmtitests_extended_nongold_1,testSCCMLTests6_1,testDDRExt_Class_0,threadMXBeanTestSuite1_0,TestAttachErrorHandling_0,cmdLineTester_dumpromclasstests_0,testSCCMLTests2_0, but TKG only printed openJcePlusTests_0

link

[2025-07-12T19:48:21.018Z] TEST TARGETS SUMMARY
[2025-07-12T19:48:21.018Z] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[2025-07-12T19:48:21.018Z] FAILED test targets:
[2025-07-12T19:48:21.018Z] 	openJcePlusTests_0
[2025-07-12T19:48:21.018Z] 
[2025-07-12T19:48:21.018Z] TOTAL: 1   EXECUTED: 1   PASSED: 0   FAILED: 1   DISABLED: 0   SKIPPED: 0
[2025-07-12T19:48:21.018Z] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

internal issue