Update github workflow to upload test coverage reports

[Chralt98]

Fixes #263.

https://blog.rust-lang.org/inside-rust/2020/11/12/source-based-code-coverage.html

[Chralt98]

@sea212 @lsaether Is it okay to use and sign up on https://docs.codecov.com/docs#step-1-sign-up-for-codecov ? I have no admin access to the repository to integrate Codecov.

If you are signing up via GitHub, you may need to request access from an admin to authorize Codecov as a third-party application. For more information see GitHub OAuth Admin Authorization or follow our video guide.

[codecov-commenter]

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@48ba93b). Click here to learn what that means. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #759   +/-   ##
=======================================
  Coverage        ?   92.23%           
=======================================
  Files           ?       80           
  Lines           ?    14062           
  Branches        ?        0           
=======================================
  Hits            ?    12970           
  Misses          ?     1092           
  Partials        ?        0           

Flag	Coverage Δ
tests	92.23% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[maltekliemann]

Just a dumb question: The report says 92.37% coverage. What exactly does that number mean? There seem to be two different workflows running, one for unit tests and one for fuzz tests, so I would have expected at least two numbers... right?

Also, doesn't this number seem surprisingly high? Can we get a complete list of lines of code reached/not reached?

[Chralt98]

Does CodeCov still need access to our repository?

That's a good question. I don't really know. I would assume no, because Codecov could comment on this PR, so I think it has been already added by Logan.

[Chralt98]

The report says 92.37% coverage. What exactly does that number mean?

@maltekliemann I did use the lcov file and produced a html website out of it. And correct me if I am wrong, but I think we reached a code coverage of that level for our tests. I try to get another message from codecov, in which you can click on the report to see the code coverage for the different crates. The current message of Codecov is not correct at the moment. I wonder if this happens, when codecov is not initialised on the main branch. I need investigation time for that.

I would assume, that we need codecov on main, because of the error message from the codecov comment.

No coverage uploaded for pull request base (main@0e27079).

[Chralt98]

@lsaether @sea212 Codecov is currently requested from me for @zeitgeistpm. Hope you accept the request to allow automatic code coverage reports.

[Chralt98]

@mkl

The report says 92.37% coverage. What exactly does that number mean?

The problem from the Codecov report is, that there exists no codecov report for the main branch. That's why we can't see the diff for the message. I also realised, that the Codecov message updates (with github edits). It does not come again as a Github message.

There seem to be two different workflows running, one for unit tests and one for fuzz tests, so I would have expected at least two numbers... right?

That's right. That is real problem. I need to fix that. You are right, we should see two codecov reports.

Also, doesn't this number seem surprisingly high? Can we get a complete list of lines of code reached/not reached?

This number seems correct.

Complete list here https://codecov.io/gh/zeitgeistpm/zeitgeist/tree/c1d8c5bd9eb2ee86413843aa3c55b4e789557bb9

https://app.codecov.io/gh/zeitgeistpm/zeitgeist/tree/chralt98-code-coverage

[Chralt98]

https://rust-fuzz.github.io/book/cargo-fuzz/coverage.html

Fuzz test coverage is extremely slow with the current version. I don't know how to solve this.

The fuzz coverage is outdated. Unfortunately I can't append my own compiler flags to get the result of code coverage for fuzz tests. https://github.com/rust-fuzz/cargo-fuzz/issues/320

For the fuzz tests, the proposed way of cargo-fuzz always produces an empty lcov file. So the coverage doesn't seem to work at least for fuzz tests.

When I run cargo fuzz coverage --release --fuzz-dir zrml/swaps/fuzz pool_join for each fuzz target, then it takes extremely long to finish, because each corpus element runs. Even in release mode it takes so long.

For the normal tests I think the result is reasonable. I know, that there are some bugs like you Malte found out, but I think it's the fault of the current Substrate environment. At least we can detect paths, which are not covered at all from the tests. That's useful.

[Chralt98]

I tried https://github.com/taiki-e/cargo-llvm-cov out @vivekvpandya . It had exactly the same result as we got with the current solution. Thanks for your investigation! I think it's fine as it is. With using cargo-llvm-cov (because it's a wrapper around cargo test) it's not so clear what actually happens (so the future maintainers would need to look up cargo-llsm-cov to understand that it's basically cargo test with code coverage and grcov).

[Chralt98]

We got a slightly better time for the current fuzz workflow. It's now 56 minutes instead of on the main branch 66 minutes. This is because of the --release profile for the swaps fuzz tests. Just in case if somebody asks about if compile time with dev (debug) beats the speed of --release.