Evaluation Removal of TAP plugin due to Job status interference

[sophia-guo]

This is to propose a discussion around whether we should continue using the TAP plugin( TapPublisher ) in our Jenkins pipeline.

Initially Tap plugin Tap plugin was introduced to scans and creates views for the test results and help to set job status ( If failed tests found, job can be set as unstable). With parallel it can also create the aggregated view at the parent level. e.g. one line summary as 11 files | 97 tests, 48 ok, 6 not ok, 43 skipped, 0 ToDo, 0 Bail Out!

For example 11 files | 97 tests, 48 ok, 6 not ok, 43 skipped, 0 ToDo, 0 Bail Out! All other information is just the Tap files themselves, which are artifacts of the jenkins job. from https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_aarch64_linux/226/ https://ci.adoptium.net/view/Test_openjdk/job/Test_openjdk21_hs_extended.openjdk_aarch64_mac/94/

Reason for concern

• We have our own strategy to set single test job status based on Tapfile results checkTestResults(results) , no need it to set job status
• Tap plugin forcibly sets job status to UNSATBLE if any test failures are found - even when wrapped in catchError or similar pipeline constructs (failedTestsMarkBuildAsFailure : boolean won't work with test failures as unstable). This makes it difficult to accurately reflect overall pipeline success or failure, especially when feature of the rerun unstable or failed jobs is enabled. For example if rerun job status is succeed we should set job as SUCCEED, even if the former run is unstable or failure. But tap plugin will set job as UNSTABLE as there is test failures in the form run of TAP file.
• The aggregated information or numbers has to be taken carefully. For example 226 the failed jobs are failed after test run, which already run all tests so testlist_0, testList_3, testList_5 are duplicated. In other cases those rerun failed job may not have tap files at all, which means this aggregating view can distort the reporting.

Proposal:

• Removing the Tap plugin
• Use junit only, which is more flexible with test result handling (e.g., skipMarkingBuildUnstable) https://github.com/adoptium/aqa-tests/issues/6238

Block https://github.com/adoptium/aqa-tests/issues/6000