Need to clarify semantic of Optional GHA Runners

[tarilabs]

Some of the Optional GHA TCK Runners, ie the Vendor specific ones, are completing Successfully (✅ ) even when the test case is clearly wrong.

This could be confusing for:

• Users looking at the GitHub and seeing GHA reports
• Users running the project locally (and not checking the generated tck_results.csv content in the results folder)

as they would see the green ✅ and potentially assume all the tests are passing.

This can be demonstrated with:

• https://github.com/dmn-tck/tck/pull/469

While the PR passes vendor-neutral Validation check (both .dmn and .xml files are semantically valid), the test should Fail as it's clearly a wrong assertion; none of the optional runners reports in tck_results.csv a success, yet some of the optional runners are green ✅ when running on GitHub or locally with Maven:

Details

I realised this, as I was working on a separate PR here: https://github.com/dmn-tck/tck/pull/468

I believe we have a common agreement that the tck_results.csv can be used to report IGNORED, not supported, and missing cases; this is very helpful when a Vendor wants to submit refreshed results and being explicit that a specific test case might relate to a feature not-yet-supported, that might eventually be supported, etc. beyond simply reporting the error.

On other hand, the semantic of the JUnit running with each test case, I believe should be the Java one: either a test case is preventively marked as ignored, or it should Fail if the test is Failing for whatever reason.

Any solution to this issue, would need careful considerations, also in order to strike a balance between: Vendors who can and are willing to share end-to-end open source solutions (both engine and runners), and Vendors who cannot share in full end-to-end open source solutions.

In my opinion, the current semantic of some of the runners (demonstrated by https://github.com/dmn-tck/tck/pull/469) that if a test is Failing, it is automatically marked as Ignored instead, and NO JUnit failures, imho:

• is potentially misleading for Users
• defies the purpose of having the runner as an Optional GHA Runner (will always be green!)

We have been using Optional GHA Runner as an helpful tool to catch potential problems, or difference of interpretation in the spec; but if a GHA runner is green because automatically assuming any Failure as Ignored, defies this rationale. At the risk of saying something tautological, just to be clear: Optional GHA Runners, as the name implies, are opt-in, and no Vendor is required to enlist the runner provided to the TCK in the list of GHA Runners.

I am happy to hear feedback both on my analysis if I missed to consider anything, and to potential solutions.