Coverage statistics do not (or cannot be configured to) account for code generated by `build.rs` into `$OUT_DIR`

[c-to-the-l]

Describe the Bug

Possibly this is my own misunderstanding of features, but I (and friends) have scoured the documentation and cannot see how to approach this problem. If I've missed something in the documentation, I can only apologise in advance.

For projects that generate code using build.rs and output generated code into $OUT_DIR (as recommended by the Cargo book), generated code is not/cannot be included in the coverage statistics for that project.

Generating code directly into src/ can fix the issue, however this is not considered good rust practice, and it would be better if there was some way to specify additional source directories to be considered for coverage.

In the example below, code generators output code into the directory specified by the env variable $OUT_DIR, which is the "correct" location for intermediate files in the build process, however it is not possible to (or not clear how to) have the tarpaulin coverage account for these files.

To reproduce

Here I will provide two examples, which produce different results, despite being ostensibly based on the same code -

Code generator, incomplete code statistics

Using the following example code, where build.rs produces code that is then included by main.rs and used by tests:

// src/main.rs

include!(concat!(env!("OUT_DIR"), "/hello.rs"));

fn main() {
    is_tested();
    println!("Hello, world!");
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn it_works() {
        assert_eq!(is_tested(), 20);
    }
}

// build.rs

use std::env;
use std::fs;
use std::path::Path;

fn main() {
    let out_dir = env::var_os("OUT_DIR").unwrap();
    let dest_path = Path::new(&out_dir).join("hello.rs");
    fs::write(
        &dest_path,
        "pub fn not_tested() {
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
            println!(\"This code isn't tested!\");
        }
        
        pub fn is_tested() -> i64 {
            println!(\"This code is tested!\");
            20
        }
        "
    ).unwrap();
    println!("cargo:rerun-if-changed=build.rs");
}

Produces the following tarpaulin output:

$ cargo tarpaulin
Feb 15 11:05:17.449  INFO cargo_tarpaulin::config: Creating config
Feb 15 11:05:17.458  INFO cargo_tarpaulin: Running Tarpaulin
Feb 15 11:05:17.458  INFO cargo_tarpaulin: Building project
Feb 15 11:05:17.458  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling tarpaulin-issue-example v0.1.0 (/home/-/tarpaulin-issue-example)
    Finished test [unoptimized + debuginfo] target(s) in 0.58s
Feb 15 11:05:18.083  INFO cargo_tarpaulin::process_handling::linux: Launching test
Feb 15 11:05:18.083  INFO cargo_tarpaulin::process_handling: running /home/-/tarpaulin-issue-example/target/debug/deps/tarpaulin_issue_example-f766830d3e522eda

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

Feb 15 11:05:18.249  INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/main.rs: 2/5
|| 
40.00% coverage, 2/5 lines covered

Code in-line, complete code statistics

// src/main.rs

pub fn not_tested() {
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
    println!("This code isn't tested!");
}

pub fn is_tested() -> i64 {
    println!("This code is tested!");
    20
}

fn main() {
    is_tested();
    println!("Hello, world!");
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn it_works() {
        assert_eq!(is_tested(), 20);
    }
}

$ cargo tarpaulin
Feb 15 11:24:06.408  INFO cargo_tarpaulin::config: Creating config
Feb 15 11:24:06.638  INFO cargo_tarpaulin: Running Tarpaulin
Feb 15 11:24:06.638  INFO cargo_tarpaulin: Building project
Feb 15 11:24:06.638  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling tarpaulin-issue-example v0.1.0 (/home/-/tarpaulin-issue-example)
    Finished test [unoptimized + debuginfo] target(s) in 0.77s
Feb 15 11:24:07.457  INFO cargo_tarpaulin::process_handling::linux: Launching test
Feb 15 11:24:07.457  INFO cargo_tarpaulin::process_handling: running /home/-/tarpaulin-issue-example/target/debug/deps/tarpaulin_issue_example-f766830d3e522eda

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

Feb 15 11:24:07.633  INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/main.rs: 4/18
|| 
22.22% coverage, 4/18 lines covered

In this second example, you can see that tarpaulin correctly picks up all 18 lines of code, whereas the first example only states 5 lines of code, of which 2 lines are tested.

Expected Behaviour

Expected behaviour would be that the % coverage and lines covered would be similar or identical for the two examples above, or that it would be possible to enable this behaviour via a flag or configuration parameter in tarpaulin.toml.

[xd009642]

So usually people are using 3rd party libraries like prost-build to generate code into the OUT_DIR which is covered by that libraries tests and don't want it included in the results, especially if it's fickle to changes between compilation or unformated code dumped onto a single line.

Also, previous compilations may have a different OUT_DIR that doesn't get cleaned up meaning those files could be picked up erroneously by tarpaulin and added to results. I could add a flag with something like --include-generated-code to allow these to be included into results, the issue then is handling stale OUT_DIRs versus ones that are there from previous builds and aren't regenerated :thinking:.

Also PRs are welcome and I'll happily provide guidance as I'm not sure when I'll get round to working on this :eyes: