jdk_security failed with 3 testcases but TEST TARGETS SUMMARY didn't grab it correctly and tap file is not correctly generated.

[sophia-guo]

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0/33 jdk_security3_0 failed with 3 testcases but TEST TARGETS SUMMARY didn't grab it correctly. It's not in the skipped test, passed test nor failed test. Hence the tap file is not correctly generated. https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0/33/artifact/aqa-tests/TKG/output_16555710233694/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0.tap/view/

15:21:40  Test results: passed: 612; failed: 3
15:21:41  Report written to /export/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0/jvmtest/openjdk/report/html/report.html
15:21:41  Results written to /export/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0/aqa-tests/TKG/output_16555710233694/jdk_security3_0/work
15:21:41  Error: Some tests failed or other problems occurred.
15:21:41  
15:21:41  jdk_security3_0_FAILED
15:21:41  
15:21:41  Nothing to be done for teardown.
15:21:41  jdk_security3_0 Finish Time: Sat Jun 18 20:21:41 2022 Epoch Time (ms): 1655580101527
15:21:41  
...
16:49:31  PASSED test targets:
16:49:31  	jdk_custom_0
16:49:31  	langtools_custom_0
16:49:31  	hotspot_custom_0
16:49:31  	hotspot_compiler_0
16:49:31  	hotspot_gc_0
16:49:31  	hotspot_runtime_0
16:49:31  	hotspot_serviceability_0
16:49:31  	jdk_beans_0
16:49:31  	jdk_io_0
16:49:31  	jdk_other_0
16:49:31  	jdk_net_0
16:49:31  	jdk_nio_0
16:49:31  	jdk_security1_0
16:49:31  	jdk_security2_0
16:49:31  	jdk_security4_0
16:49:31  	jdk_text_0
16:49:31  	jdk_time_0
16:49:31  	jdk_management_0
16:49:31  	jdk_jmx_0
16:49:31  	jdk_rmi_0
16:49:31  	jdk_instrument_0
16:49:31  	jdk_imageio_0
16:49:31  
16:49:31  FAILED test targets:
16:49:31  	hotspot_jre_0
16:49:31  	jdk_tools_0

The junit test result can correctly show the failed three tests.

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0/33/testReport/

[llxia]

@renfeiw can you take a look at this from TKG respective? Thanks

[llxia]

I suspect that the test output breaks the parser in both TKG and TRSS. At least this is the case on the TRSS side. In the test, it outputted a string that is very similar to what TRSS is looking for. As result, TRSS treated it as a test:

[2022-06-18T18:55:11.265Z] Running test with provider SunPKCS11-nss (security manager disabled) ...

https://trss.adoptium.net/output/test?id=62ae3d4c250c3c428c2924be

[llxia]

To avoid this happening again, I would suggest changing our keywords in the test output. Instead of Running test <test target>, could we output a more specific string - i.e., Running AQAvit test target <test target> in TKG? @smlambert what do you think?

[smlambert]

Yes, changing it to the longer and more specific string is a good plan (also helps brand awareness).

[llxia]

@renfeiw and I had the discussion, instead of changing the header string, we will match multiple lines. In this case, TRSS will match the following:

===============================================
Running test <test target> ...

For details, please see TRSS PR https://github.com/adoptium/aqa-test-tools/pull/713

@renfeiw please update TKG parser as well. Thanks