spin2cpp
spin2cpp copied to clipboard
Add test coverage reporting
Having come across the odd bit of code that seems to be old bitrotted gunk, I thought it'd be neat to hook some line-level coverage reporting to the test suite. Annoyingly, it only seems to work on GCC 8 under Linux due to weirdness with where it wants to place its files, but for running it on CI that's fine, I guess. Speaking of, I've hooked it to codecov for generation of fancy statistics, which looks a bit like this: https://codecov.io/gh/Wuerfel21/spin2cpp/src/test-coverage/spinc.c If you merge this PR, I think you need to log into codecov yourself so it can access the repository.
Codecov Report
:exclamation: No coverage uploaded for pull request base (
master@1da37fc
). Click here to learn what that means. The diff coverage isn/a
.
@@ Coverage Diff @@
## master #169 +/- ##
=========================================
Coverage ? 53.91%
=========================================
Files ? 57
Lines ? 33898
Branches ? 0
=========================================
Hits ? 18276
Misses ? 15622
Partials ? 0
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 1da37fc...64696fa. Read the comment docs.
Hmmm... this sounds interesting, but I'd like to dig into a bit before merging.
It is probably time to add some more tests. I've got enough of the BASIC and C runtime working with bytecode now that we can do floating point and strings, so a pretty good selection of programs should run.
It is probably time to add some more tests.
Yeah, that's the idea. Knowing what isn't covered yet is of good use in that, I think.
Though there's the difficulty that bytecode output doesn't really have an intermediate form that can be diffed. Listings don't really work that well for that since any size change causes the rest of the file to change. I think emitting a listing without the address column would work okay-ish though.
Relatedly, there probably need to be a couple more BASIC tests in general, basiclang.c
has the lowest coverage of all the frontends at ~24%
A lot of the BASIC testing is in the runtime tests. I should start adding more .bas files to the PASM and C++ backend tests too though, it would give us a bit more coverage. I'll try to make future tests in BASIC; I've tended to use Spin for the really simple cases and C for the fancy type specific things.
On a related note, I wonder if Gear or spinsim or something like that could execute the runtime tests on a PC, allowing us to do coverage testing even for the runtime tests?
Yeah, spinsim could probably be hooked up to run tests. Not sure about the speed though.