Added support for Hpc coverage per test

[agustinmista]

Hello @UnkindPartition!

I'm opening this PR to discuss a feature I added to tasty-core: Hpc coverage integration that produces individual coverage reports per test. The motivation for this is that having more granular Hpc reports allows developing some new interesting stuff, e.g., HTML coverage reports with expressions annotated by the tests exercising them (this was created using my WIP fork of Hpc).

The way this works is by wrapping the test runner using a function that clears the internal Hpc coverage vector, runs the test, reflects the Hpc vector to a Haskell value that contains the individual coverage of such test, and saves it to a .tix file. These .tix files are stored in a destination folder set by the user with the --save-tix TIX_PATH CLI option and are structured to follow the test tree hierarchy. E.g., running the following test tree:

testGroup "One" [ testGroup "Two" [ testCase "Three" ... ] ]

with --save-tix foo will create:

foo/One/Two/Three.tix

One possible drawback of this feature is that test should run sequentially so the internal Hpc coverage vector doesn't get mangled with ticks generated by different parallel tests. To account for this, using --save-tix implies NumThreads=1.

I also included the script core-tests/core-tests-hpc.sh in case you want to try it by yourself. It assumes the existence of my fork of HPC, but you can comment the last command if you don't want to produce an HTML report.

I hope you like this feature! 😄

/Agustín

[UnkindPartition]

Hi Agustín,

This sounds like a cool idea. However, I don't see why it needs to be in tasty-core. Try implementing it at as a standalone TestManager, using the API from https://hackage.haskell.org/package/tasty-1.4.2/docs/Test-Tasty-Runners.html. You wouldn't even need to worry about the parallelism, as you would be in charge of how the tests are run.

[agustinmista]

Thanks for the feedback. I will give it a try as soon as I get some free time.

In the meantime, do you have any example implemented using the TestManager API?

/Agustín

[UnkindPartition]

There are some trivial ones, like listingTests and includingOptions, that don't actually run the tests.

Yours is the first case that I can recall where you actually want to run the tests but do it differently than the standard runner. So it seems like a great fit, but there don't seem to be any prior examples.

[Bodigrim]

@agustinmista I second the idea of implementing this as a separate test ingredient, outside of tasty-core. Unless there are specific objections voiced within two weeks, I intend to close the PR.

[agustinmista]

Hi @Bodigrim!

I agree that it would be better to implement this feature as a custom ingredient. I will close this PR.