rust-lang
Coverage
Figure out a way to run gcov on miri, so we know if we aren't testing something or if some hack becomes obsolete
I think this is relatively doable nowadays, as a far bit of infrastructure in this area has stabilized:
Instrumentation-based Code Coverage
The LLVM coverage data in particular is pretty nice.
Might be a good first issue now?
Maybe not a good "first" issue, since I assume it will involve some fighting with the build system. But yeah seems worth promoting this a bit so maybe someone can look into it.
What is not clear at all to me is how to best get this into CI. Ideally we'd somehow know when a line that used to be covered is no longer covered, but that requires keeping state across CI runs... not sure how that would work / what people usually do here.
I would be interested in trying this. I would need to do some research first though. I haven't done anything similar before.
Yeah, definitely -- we'd want to know what is even possible in this space and what the final result would look like before investing into any kind of polished solution.
I finally found some time to work on this. Just by adding RUSTFLAGS="-C instrument-coverage" in the ./miri test command:
[mandragore@archibald miri]$ git diff
diff --git a/miri-script/src/util.rs b/miri-script/src/util.rs
index f5a6a8188..45f587dfa 100644
--- a/miri-script/src/util.rs
+++ b/miri-script/src/util.rs
@@ -188,7 +188,7 @@ impl MiriEnv {
}
pub fn test(&self, crate_dir: impl AsRef<OsStr>, args: &[String]) -> Result<()> {
- self.cargo_cmd(crate_dir, "test").args(args).run()?;
+ self.cargo_cmd(crate_dir, "test").args(args).env("RUSTFLAGS", "-C instrument-coverage").run()?;
Ok(())
}
I was able to generate the profraw files. Then after install cargo-binutils and running:
[root@e4aaf2fc8c9d miri]# cargo-profdata -- merge -sparse default_*.profraw -o cooked.profdata
[root@e4aaf2fc8c9d miri]# cargo-cov -- report --instr-profile=cooked.profdata --object target/debug/miri -ignore-filename-regex="root/.cargo/registry/*|root/.rustup/*|/rust/deps/*|target/*"
(the root in the regex above is because I use a container to run tests)
I got this report:
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
alloc_addresses/mod.rs 147 16 89.12% 24 4 83.33% 303 7 97.69% 0 0 -
alloc_addresses/reuse_pool.rs 44 0 100.00% 6 0 100.00% 93 0 100.00% 0 0 -
alloc_bytes.rs 43 8 81.40% 15 4 73.33% 70 6 91.43% 0 0 -
bin/miri.rs 331 115 65.26% 37 19 48.65% 523 164 68.64% 0 0 -
borrow_tracker/mod.rs 123 10 91.87% 37 1 97.30% 288 15 94.79% 0 0 -
borrow_tracker/stacked_borrows/diagnostics.rs 176 18 89.77% 42 1 97.62% 342 27 92.11% 0 0 -
borrow_tracker/stacked_borrows/item.rs 26 1 96.15% 8 0 100.00% 46 1 97.83% 0 0 -
borrow_tracker/stacked_borrows/mod.rs 406 37 90.89% 44 1 97.73% 666 41 93.84% 0 0 -
borrow_tracker/stacked_borrows/stack.rs 157 2 98.73% 18 0 100.00% 270 1 99.63% 0 0 -
borrow_tracker/tree_borrows/diagnostics.rs 193 12 93.78% 33 1 96.97% 382 18 95.29% 0 0 -
borrow_tracker/tree_borrows/mod.rs 221 37 83.26% 28 1 96.43% 410 29 92.93% 0 0 -
borrow_tracker/tree_borrows/perms.rs 148 31 79.05% 34 5 85.29% 227 40 82.38% 0 0 -
borrow_tracker/tree_borrows/tree.rs 224 30 86.61% 48 5 89.58% 521 30 94.24% 0 0 -
borrow_tracker/tree_borrows/unimap.rs 60 23 61.67% 28 11 60.71% 129 51 60.47% 0 0 -
clock.rs 37 1 97.30% 10 0 100.00% 72 1 98.61% 0 0 -
concurrency/cpu_affinity.rs 23 6 73.91% 7 0 100.00% 54 9 83.33% 0 0 -
concurrency/data_race.rs 622 69 88.91% 102 1 99.02% 1106 24 97.83% 0 0 -
concurrency/init_once.rs 47 47 0.00% 10 10 0.00% 77 77 0.00% 0 0 -
concurrency/range_object_map.rs 38 1 97.37% 11 0 100.00% 61 1 98.36% 0 0 -
concurrency/sync.rs 258 46 82.17% 56 5 91.07% 559 20 96.42% 0 0 -
concurrency/thread.rs 377 59 84.35% 97 14 85.57% 695 78 88.78% 0 0 -
concurrency/vector_clock.rs 119 57 52.10% 33 6 81.82% 234 101 56.84% 0 0 -
concurrency/weak_memory.rs 184 20 89.13% 23 0 100.00% 336 8 97.62% 0 0 -
diagnostics.rs 264 27 89.77% 18 2 88.89% 487 42 91.38% 0 0 -
eval.rs 159 47 70.44% 10 2 80.00% 315 47 85.08% 0 0 -
helpers.rs 547 145 73.49% 104 24 76.92% 849 155 81.74% 0 0 -
intrinsics/atomic.rs 219 53 75.80% 14 0 100.00% 215 10 95.35% 0 0 -
intrinsics/mod.rs 499 131 73.75% 4 0 100.00% 315 14 95.56% 0 0 -
intrinsics/simd.rs 910 242 73.41% 6 0 100.00% 590 56 90.51% 0 0 -
machine.rs 503 98 80.52% 89 7 92.13% 1005 103 89.75% 0 0 -
mono_hash_map.rs 40 4 90.00% 14 2 85.71% 60 8 86.67% 0 0 -
operator.rs 74 15 79.73% 8 0 100.00% 85 4 95.29% 0 0 -
provenance_gc.rs 89 14 84.27% 27 6 77.78% 141 22 84.40% 0 0 -
range_map.rs 80 14 82.50% 14 3 78.57% 147 21 85.71% 0 0 -
shims/alloc.rs 102 20 80.39% 8 0 100.00% 110 1 99.09% 0 0 -
shims/backtrace.rs 156 48 69.23% 6 0 100.00% 167 9 94.61% 0 0 -
shims/env.rs 61 21 65.57% 10 2 80.00% 72 21 70.83% 0 0 -
shims/extern_static.rs 54 26 51.85% 4 0 100.00% 63 11 82.54% 0 0 -
shims/foreign_items.rs 911 300 67.07% 18 3 83.33% 688 114 83.43% 0 0 -
shims/io_error.rs 96 43 55.21% 15 3 80.00% 110 32 70.91% 0 0 -
shims/native_lib.rs 99 43 56.57% 6 0 100.00% 117 34 70.94% 0 0 -
shims/os_str.rs 115 90 21.74% 24 16 33.33% 234 160 31.62% 0 0 -
shims/panic.rs 102 38 62.75% 7 0 100.00% 139 16 88.49% 0 0 -
shims/time.rs 176 106 39.77% 16 11 31.25% 289 136 52.94% 0 0 -
shims/tls.rs 145 77 46.90% 16 7 56.25% 213 99 53.52% 0 0 -
shims/unix/android/foreign_items.rs 12 12 0.00% 2 2 0.00% 20 20 0.00% 0 0 -
shims/unix/env.rs 203 56 72.41% 17 1 94.12% 207 15 92.75% 0 0 -
shims/unix/fd.rs 264 93 64.77% 53 9 83.02% 450 105 76.67% 0 0 -
shims/unix/foreign_items.rs 1342 593 55.81% 8 3 62.50% 677 116 82.87% 0 0 -
shims/unix/freebsd/foreign_items.rs 81 81 0.00% 2 2 0.00% 61 61 0.00% 0 0 -
shims/unix/fs.rs 1017 562 44.74% 71 29 59.15% 1252 504 59.74% 0 0 -
shims/unix/linux/epoll.rs 275 73 73.45% 24 4 83.33% 418 39 90.67% 0 0 -
shims/unix/linux/eventfd.rs 65 16 75.38% 6 1 83.33% 135 6 95.56% 0 0 -
shims/unix/linux/foreign_items.rs 192 69 64.06% 2 0 100.00% 128 12 90.62% 0 0 -
shims/unix/linux/mem.rs 38 11 71.05% 1 0 100.00% 49 2 95.92% 0 0 -
shims/unix/linux/sync.rs 123 47 61.79% 1 0 100.00% 126 30 76.19% 0 0 -
shims/unix/macos/foreign_items.rs 249 249 0.00% 2 2 0.00% 149 149 0.00% 0 0 -
shims/unix/macos/sync.rs 58 58 0.00% 7 7 0.00% 68 68 0.00% 0 0 -
shims/unix/mem.rs 88 33 62.50% 2 0 100.00% 119 20 83.19% 0 0 -
shims/unix/solarish/foreign_items.rs 85 85 0.00% 2 2 0.00% 74 74 0.00% 0 0 -
shims/unix/sync.rs 476 128 73.11% 44 0 100.00% 576 28 95.14% 0 0 -
shims/unix/thread.rs 74 19 74.32% 7 0 100.00% 88 1 98.86% 0 0 -
shims/unix/unnamed_socket.rs 144 42 70.83% 9 1 88.89% 264 36 86.36% 0 0 -
shims/wasi/foreign_items.rs 22 22 0.00% 2 2 0.00% 26 26 0.00% 0 0 -
shims/windows/env.rs 147 147 0.00% 11 11 0.00% 172 172 0.00% 0 0 -
shims/windows/foreign_items.rs 773 773 0.00% 6 6 0.00% 538 538 0.00% 0 0 -
shims/windows/handle.rs 54 54 0.00% 12 12 0.00% 100 100 0.00% 0 0 -
shims/windows/sync.rs 139 139 0.00% 7 7 0.00% 159 159 0.00% 0 0 -
shims/windows/thread.rs 56 56 0.00% 2 2 0.00% 58 58 0.00% 0 0 -
shims/x86/aesni.rs 99 28 71.72% 6 0 100.00% 120 1 99.17% 0 0 -
shims/x86/avx.rs 330 86 73.94% 1 0 100.00% 182 12 93.41% 0 0 -
shims/x86/avx2.rs 481 144 70.06% 1 0 100.00% 241 13 94.61% 0 0 -
shims/x86/bmi.rs 64 11 82.81% 1 0 100.00% 70 2 97.14% 0 0 -
shims/x86/gfni.rs 131 33 74.81% 3 0 100.00% 105 1 99.05% 0 0 -
shims/x86/mod.rs 1072 309 71.18% 39 1 97.44% 814 28 96.56% 0 0 -
shims/x86/sha.rs 120 30 75.00% 13 0 100.00% 135 1 99.26% 0 0 -
shims/x86/sse.rs 198 52 73.74% 2 0 100.00% 114 7 93.86% 0 0 -
shims/x86/sse2.rs 334 91 72.75% 2 0 100.00% 193 8 95.85% 0 0 -
shims/x86/sse3.rs 34 9 73.53% 1 0 100.00% 31 2 93.55% 0 0 -
shims/x86/sse41.rs 142 44 69.01% 1 0 100.00% 97 3 96.91% 0 0 -
shims/x86/sse42.rs 401 115 71.32% 7 0 100.00% 249 14 94.38% 0 0 -
shims/x86/ssse3.rs 132 43 67.42% 1 0 100.00% 76 4 94.74% 0 0 -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 18920 6661 64.79% 1567 281 82.07% 22216 4299 80.65% 0 0 -
I guess we can create a coverage command for the miri script to automate the above? Or a flag in the test command?
I created this draft PR #3954
It requires that cargo-binutils is installed to generated the reports. It's so that we use the correct llvm version, compatible with what rustc is using. There is also llvm_profparser that tarpaulin is using. Maybe we can use that if we want to generate more complicated reports. But as a first step maybe the PR is good enough as it is.