github action check status is "neutral" when tests pass

[gotchipete]

Bug description

When all tests pass, github still has the action marked as "neutral" instead of a pass / green check mark

How to reproduce

Run the test runner with a suite of passing tests, observe a neutral result (see screenshot):

• Expected behavior

Passing tests should show a green check mark for the github action

My action code:

      - uses: game-ci/unity-test-runner@v2
        id: tests
        env:
          SOME_ENV_VAR: ${{ secrets.SOME_SECRET }}
        with:
          projectPath: ./apps/foobar
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          coverageOptions: "generateAdditionalMetrics;generateHtmlReport;generateBadgeReport;generateAdditionalReports"
      

[davidmfinol]

Personally, I like to refer to the status of the job that runs the unit tests to see if tests have passed or not. I also like that the unit test results show as neutral, so that they are easy to find and know that they are separate from the other jobs.

EDIT: I've marked this as wontfix for now, but I think we should keep this bug open to hear others' thoughts on this.

[gotchipete]

@davidmfinol ah, so a way to get a green checkmark on test success is to split out the test step into its own job? Will the job actually report a fail or success in this case, since the step will still be reporting neutral? Guess I'll try it and see.

Also, I don't agree with the sentiment that the results showing neutral for "findability" is a good thing. Marking a step "neutral" when it is 100% clear as to its pass or fail status (tests passed, or one or more tests failed, respectively) just so it can be found more easily in a list seems to be a complete misuse of the property.

[timcassell]

You could use a different test reporter to show the green checkmark, like dorny/test-reporter. See comments in #142 for details.

[gotchipete]

You could use a different test reporter to show the green checkmark, like dorny/test-reporter. See comments in #142 for details.

Ahh I was looking for one, thank u @timcassell !

[davidmfinol]

Will the job actually report a fail or success in this case, since the step will still be reporting neutral?

Yep, I look at the status of the test job to see if that job actually passed or failed. I wouldn't call the test report a "step". It's just additional information that supplements the actual step that is the test job.

And like @timcassell suggests you could also use a different test reporter, or you could fork this repo and have your fork modify https://github.com/game-ci/unity-test-runner/blob/31086d985910613d75c32ba965f657df9c298820/src/model/results-check.ts#L91 to change the conclusion from neutral to success.

[gotchipete]

Ok, thanks. I'm looking for the step to report pass/fail b/c I simply want to fail the build and disallow a merge if test fail. Seems like a fairly common use case to me 🤔 ... but I'll look at the reporter or forking. Thanks for the work on everything else that makes this work!

On Mon, May 15, 2023 at 12:10 PM David Finol @.***> wrote:

Will the job actually report a fail or success in this case, since the step will still be reporting neutral?

Yep, I look at the status of the test job to see if that job actually passed or failed. I wouldn't call the test report a "step". It's just additional information that supplements the actual step that is the test job.

And like @timcassell https://github.com/timcassell suggests you could also use a different test reporter, or you could fork this repo and have your fork modify https://github.com/game-ci/unity-test-runner/blob/31086d985910613d75c32ba965f657df9c298820/src/model/results-check.ts#L91 to change the conclusion from neutral to success.

— Reply to this email directly, view it on GitHub https://github.com/game-ci/unity-test-runner/issues/220#issuecomment-1548151982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AW7WD76PCDQLDASNJ4F5DA3XGJIO3ANCNFSM6AAAAAAX7YIWVY . You are receiving this because you authored the thread.Message ID: @.***>