microsoft
ci: Add code coverage
This PR enables code coverage for tests and adds the logic to upload coverage data to CodeCov.
Scope
Coverage is currently limited only to tests in the top level workspace. It does not include tests from the tests folder because they primarily verify macro expansion which is hard to get coverage for. We could focus on them at a later point.
Also we have enabled coverage only for x64 because Windows arm64 is currently not supported by cargo-llvm-cov, which is the tool we are using.
Changes
Apart from changing the test.yaml GitHub action, I also had to make some changes in cargo-wdk tests. It was to make sure Cargo and Rust environment variables from the cargo llvm-cov process do not get inherited by cargo wdk build process because they can interfere with its behaviour. For instance, cargo llvm-cov sets an environment variable to enable instrumentation in the code being built. If it is allowed to be inherited by cargo wdk build instrumentation will be enabled for the test driver projects being built as well which will impact the test.
This argument can be generalized and extended to any Cargo or Rust variables. Therefore, before calling cargo-wdk in the tests we now make sure all Cargo and Rust variables are removed from its environment.
Results
To see what code coverage will look like after this PR, see this example from an earlier run.
Closes https://github.com/microsoft/windows-drivers-rs/issues/431
Welcome to Codecov :tada:
Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.
Thanks for integrating Codecov - We've got you covered :open_umbrella:
I want to make sure we're aligned on the goal here. I think what we want is:
- see overall coverage from tests
- see diff in coverage from tests in PR
- block pr if coverage below some value
- be able to run code coverage evaluation locally
Does this accomplish all of this, or does the tool used in this pr have the ability to accomplish that?
Other questions I have:
- Can we run the action/tool multiple times and combine the results? or at least see the results from all the runs?
- to see unit test vs integration test coverage
- to be able to run multiple times with different wdk configurations active and combine results (or at least be able to see all the results)
- Is the line coverage, branch coverage, or Modified Condition/Decision coverage?
I want to make sure we're aligned on the goal here. I think what we want is:
- see overall coverage from tests
- see diff in coverage from tests in PR
- block pr if coverage below some value
- be able to run code coverage evaluation locally
Does this accomplish all of this, or does the tool used in this pr have the ability to accomplish that?
Other questions I have:
Can we run the action/tool multiple times and combine the results? or at least see the results from all the runs?
- to see unit test vs integration test coverage
- to be able to run multiple times with different wdk configurations active and combine results (or at least be able to see all the results)
Is the line coverage, branch coverage, or Modified Condition/Decision coverage?
Code coverage implemented here relies on two tools: cargo-llvm-cov and Codecov. cargo-llvm-cov is a drop-in replacement for cargo test that internally invokes cargo test to run the tests and then emits coverage. Codecov is an online visualization service. The typical workflow is to generate a coverage report with cargo-llvm-cov and upload it to Codecov which is what we're doing in test.yaml.
Given that background, here are the answers to your questions:
- see overall coverage from tests
The report will contain coverage for whatever tests we run with cargo llvm-cov. Since we running all the workspace tests you will see coverage for all of them in Codecov.
- see diff in coverage from tests in PR
Codecov can show percentage change in coverage on the PR page itself and allows comparing coverage across commits. It also shows a graph of coverage changes over time.
- block pr if coverage below some value
Yes, that is possible.
- be able to run code coverage evaluation locally
You can run cargo llvm-cov locally and it will print coverage on stdout
Can we run the action/tool multiple times and combine the results? or at least see the results from all the runs?
- to see unit test vs integration test coverage
- to be able to run multiple times with different wdk configurations active and combine results (or at least be able to see all the results)
Codecov automatically merges uploads from multiple runs into a single report. But I am not sure if it can show separate coverage for individual test groups like unit and integration tests.
- Is the line coverage, branch coverage, or Modified Condition/Decision coverage?
cargo-llvm-cov supports line and region coverage. Branch coverage is supported but is unstable and requires nightly. We will enable it when it becomes stable.