tarpaulin
tarpaulin copied to clipboard
Coverage statistics do not (or cannot be configured to) account for code generated by `build.rs` into `$OUT_DIR`
Describe the Bug
Possibly this is my own misunderstanding of features, but I (and friends) have scoured the documentation and cannot see how to approach this problem. If I've missed something in the documentation, I can only apologise in advance.
For projects that generate code using build.rs
and output generated code into $OUT_DIR
(as recommended by the Cargo book), generated code is not/cannot be included in the coverage statistics for that project.
Generating code directly into src/
can fix the issue, however this is not considered good rust practice, and it would be better if there was some way to specify additional source directories to be considered for coverage.
In the example below, code generators output code into the directory specified by the env variable $OUT_DIR
, which is the "correct" location for intermediate files in the build process, however it is not possible to (or not clear how to) have the tarpaulin coverage account for these files.
To reproduce
Here I will provide two examples, which produce different results, despite being ostensibly based on the same code -
Code generator, incomplete code statistics
Using the following example code, where build.rs
produces code that is then included by main.rs
and used by tests:
// src/main.rs
include!(concat!(env!("OUT_DIR"), "/hello.rs"));
fn main() {
is_tested();
println!("Hello, world!");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn it_works() {
assert_eq!(is_tested(), 20);
}
}
// build.rs
use std::env;
use std::fs;
use std::path::Path;
fn main() {
let out_dir = env::var_os("OUT_DIR").unwrap();
let dest_path = Path::new(&out_dir).join("hello.rs");
fs::write(
&dest_path,
"pub fn not_tested() {
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
println!(\"This code isn't tested!\");
}
pub fn is_tested() -> i64 {
println!(\"This code is tested!\");
20
}
"
).unwrap();
println!("cargo:rerun-if-changed=build.rs");
}
Produces the following tarpaulin output:
$ cargo tarpaulin
Feb 15 11:05:17.449 INFO cargo_tarpaulin::config: Creating config
Feb 15 11:05:17.458 INFO cargo_tarpaulin: Running Tarpaulin
Feb 15 11:05:17.458 INFO cargo_tarpaulin: Building project
Feb 15 11:05:17.458 INFO cargo_tarpaulin::cargo: Cleaning project
Compiling tarpaulin-issue-example v0.1.0 (/home/-/tarpaulin-issue-example)
Finished test [unoptimized + debuginfo] target(s) in 0.58s
Feb 15 11:05:18.083 INFO cargo_tarpaulin::process_handling::linux: Launching test
Feb 15 11:05:18.083 INFO cargo_tarpaulin::process_handling: running /home/-/tarpaulin-issue-example/target/debug/deps/tarpaulin_issue_example-f766830d3e522eda
running 1 test
test tests::it_works ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
Feb 15 11:05:18.249 INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/main.rs: 2/5
||
40.00% coverage, 2/5 lines covered
Code in-line, complete code statistics
// src/main.rs
pub fn not_tested() {
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
println!("This code isn't tested!");
}
pub fn is_tested() -> i64 {
println!("This code is tested!");
20
}
fn main() {
is_tested();
println!("Hello, world!");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn it_works() {
assert_eq!(is_tested(), 20);
}
}
$ cargo tarpaulin
Feb 15 11:24:06.408 INFO cargo_tarpaulin::config: Creating config
Feb 15 11:24:06.638 INFO cargo_tarpaulin: Running Tarpaulin
Feb 15 11:24:06.638 INFO cargo_tarpaulin: Building project
Feb 15 11:24:06.638 INFO cargo_tarpaulin::cargo: Cleaning project
Compiling tarpaulin-issue-example v0.1.0 (/home/-/tarpaulin-issue-example)
Finished test [unoptimized + debuginfo] target(s) in 0.77s
Feb 15 11:24:07.457 INFO cargo_tarpaulin::process_handling::linux: Launching test
Feb 15 11:24:07.457 INFO cargo_tarpaulin::process_handling: running /home/-/tarpaulin-issue-example/target/debug/deps/tarpaulin_issue_example-f766830d3e522eda
running 1 test
test tests::it_works ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
Feb 15 11:24:07.633 INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/main.rs: 4/18
||
22.22% coverage, 4/18 lines covered
In this second example, you can see that tarpaulin correctly picks up all 18 lines of code, whereas the first example only states 5 lines of code, of which 2 lines are tested.
Expected Behaviour
Expected behaviour would be that the % coverage
and lines covered
would be similar or identical for the two examples above, or that it would be possible to enable this behaviour via a flag or configuration parameter in tarpaulin.toml
.
So usually people are using 3rd party libraries like prost-build to generate code into the OUT_DIR
which is covered by that libraries tests and don't want it included in the results, especially if it's fickle to changes between compilation or unformated code dumped onto a single line.
Also, previous compilations may have a different OUT_DIR
that doesn't get cleaned up meaning those files could be picked up erroneously by tarpaulin and added to results. I could add a flag with something like --include-generated-code
to allow these to be included into results, the issue then is handling stale OUT_DIR
s versus ones that are there from previous builds and aren't regenerated :thinking:.
Also PRs are welcome and I'll happily provide guidance as I'm not sure when I'll get round to working on this :eyes: