Calculate Test Coverage

[molovo]

Could possibly use zprof to work out which lines are covered during the course of running a test, but this may be tricky. Something to look into.

Will probably rely on elements of #8 for reporting coverage data.

[molovo]

Proposal for coverage reporting

Line-by-line coverage simply isn't possible with ZSH, without some sort of extension to record each one of code as it is read and executed. That's simply beyond my ability at the moment, so here I'm proposing a sort of pseudo code coverage, which records coverage of functions rather than individual lines.

We can record the size of the $functions array at the start of testing, and then source all included files (in a subshell so the environment is not affected) and then recount $functions to get the number of functions defined in the project. We can then grep the output of zprof after each test to determine which functions were called by the test, and mark them as covered. Similarly to other languages, to avoid all called functions being marked as covered, we can set which functions are covered within the test. Example syntax:

@test 'My awesome test' {
  @covers _my_awesome_function 

  # test code here...
}

This would only mark _my_awesome_function as covered if it is called, and will not affect coverage of any other function. Similarly, if _my_awesome_function is not called within the test, it will not be marked as covered.

This is a little bit more vague than proper line-level coverage, but it would be fairly easy to implement and is certainly a lot better than anything we've got in ZSH at the moment.

Since coverage would only be performed with an option or if defined in .zunit.yml, I should be able to include this in v0.6.0 without any breaking changes.

One day I might look into developing a proper C extension for ZSH to do line coverage, but I'm not a C developer so it's not something I could do at the moment.

[molovo]

This isn't going to happen by v0.6.0. Even the simplified version of coverage I've proposed above is a lot more difficult to implement than I'd anticipated. It's still a feature I'd like to include, but I'm leaving it marked as future, and will hopefully get it included in a future release.