Describe how to use (or integrate) results reporter in test runner

[webbertakken]

Context Test runner currently saves logs to the artifacts folder. (since https://github.com/webbertakken/unity-test-runner/pull/37) That way we can use test results reporter (which generates a html report)

Solution Either we have to add a section to the readme, about how to run an additional action for producing the html report, or we should integrate the reporter in this action and support it natively. Generating a report doesn't sound too heavy to me.

What are your thoughts? Do we know how longit takes for the report to generate?

[Macoron]

Hey, thanks for a quick merge!

Unity Reporter distributes as compiled dotnet .dll which can be run on ubuntu by this action. I think, the most easiest way to setup CI pipeline is:

• Finish tests
• Install dotnet SDK 2.0 (or even better dotnet runtime 2.0)
• Run reporter
• Send html report as an artifact

I don't know should it be part of unity tester or separate repo. I'm kinda new in CI, so not sure which option is better. Right now I'm testing how robust Unity Reporter it actually is.

[webbertakken]

Having a look at it again; it doesn't seem like a very popular tool. Last commits were in March 2019, the readme is bloated and it only has a few stars.

https://github.com/Unity-Technologies/UnityTestRunnerResultsReporter

That said, there doesn't have to be anything wrong with it. It is probably a pretty straight forward tool. Curious to hear your findings!

[Macoron]

Yeah, I found it only because Unity Dev point it to me (see this forum thread). I tested it a bit, It kinda work, but not perfect (see the attached report).

There are some errors, missing vars, etc. And I'm not sure why they need .log file, can't see where it used in actual report. But it's a good start.

[webbertakken]

My two goto tools for coverage reporting are Coveralls and CodeCov. There are also C# solutions for Coveralls and CodeCov also with an (example).

Unfortunately both seem to use OpenCover which seems to be Windows only. Also doesn't seem to be very active anymore.

I also found coverlet which seems to be promising and offers cross-platform compatibility. In it's repository there is a thread where Unity is being mentioned too.

In that thread, there's also a visual tool being suggested, called ReportGenerator which itself hints at multiple test frameworks that offer code coverage output.

For me this is a bit out of scope, but i'm willing to accept merge requests if anyone finds a good solution that works for everyone!

[Macoron]

Finally finished a proof of concept here (you can download attached artifact and see yourself): https://github.com/Macoron/unity-test-runner/pull/3/checks?check_run_id=447950614

So it's kinda work. Though there are several problems:

• HTML output missing several keys (looks like it's because some internal Unity CI setup). Also error messages looks weird with [CDATA] tags
• Need some way to distribute binary files (compiled reporter DLL and html template). Maybe nuget will work? Or use docker? Not sure, to be honest.
• The original repo has no code license. According to github rules it adds a lot of headache (read-only, no changes, etc). Already created issue to request licence.

In ideal situation it's need to be implemented as independent dockered Github Action (not as part of unity-testrunner). I have no experience in that, so will just include it as is in my CI pipeline.

Yeah, not perfect, but a good start.

[Macoron]

Nice, I found much better solution called Extent Reports. It has very nice UI and Apache 2.0 license.

Made a few tweaks in my fork to make it run on .Net Core. And now it can generate awesome .html reports:

You can check modified workflow here. Download artifacts to see generated html. As I mentioned before, if someone will manage wrap it in dockered action - you will be a real hero.

[webbertakken]

That looks nicer indeed!

What stops you from installing it as a nuget package in the docker image?

What are the things you have considered?

[Macoron]

@webbertakken I just have zero experience in dockers. Maybe it's not that hard, but don't have dedication right now to setup properly.

[webbertakken]

This also looks to be interesting / promising: https://docs.unity3d.com/Packages/[email protected]/manual/UsingCodeCoverage.html#enable-code-coverage

[webbertakken]

@Macoron What did you end up using as your solution?

[Macoron]

Right now we use Extent Reports (described above). It works great and was easy to setup. I compiled it for .NET Core and placed binary files inside repository. Here is example of the test run and here is workflow.

Like I said before, more clever solution will be wrap it all in docker action with extent binaries and .NET inside. But this works too.

[webbertakken]

Unity has updated their codecoverage package to 0.3. See also an example implementation of it at gableroux/unity3d-gitlab-ci-example#42.

[qubitz]

Seems like extent should be a .NET Core Tool, not a regular nuget package. Regular nuget packages are meant to be written into a .csproj file and referenced in code as far as I'm aware.

If extent was a .NET Core Tool, workflows could use the actions/setup-dotnet like so:

- uses: actions/setup-dotnet@v1
  with:
    dotnet-version: '3.1.100'
- run: |
    dotnet tool install -g extent
    extent -i <test-results>.xml -o <test-results>.html

(edit) I suggested some changes to this end: Update project to .NET Core

[webbertakken]

I no longer think we should implement Extent or use it as the recommended option (even though it looks nice at first sight).

Instead we should incorporate the official coverage tool (which includes generating reports) from Unity. See the changes in linked example above.

[dyburke]

@webbertakken Hi, I'm confused as to how Unity's code coverage tool can generate test reports. I can't find anything in the documentation that really describes the same functionality that extent does; it appears the code coverage reports soley report on code coverage with no test results.

Am I missing something?

[webbertakken]

I don't think you're missing much. It's always a bit hard to find the right documentation for Unity related topics. That said, it's at version 0.3 last time I checked, so it's also understandable.

You would have to pass a few flags to the commandline, similar to what @fehrox tried to do here and similar to what Arthur Aulicino implemented here.

To me coverage reports are a lot more important than an overview which tests failed to be honest.

Hope that helps.