cargo-llvm-cov
cargo-llvm-cov copied to clipboard
Improve support for doctests
STATUS:
#40 fixed all of the errors and many of the warnings
The remaining warnings are probably the column offset errors mentioned in rust-lang/rust#79417 (comment), and it's not clear whether we can fix them on our side at this time.
This is currently available via --doctests
flag as an unstable feature.
cargo llvm-cov --doctests
Refs:
- https://doc.rust-lang.org/rustc/instrument-coverage.html#including-doc-tests
- https://github.com/rust-lang/rust/issues/79417
- https://github.com/rust-lang/rust/issues/56925
STATUS: I think it's better than before, but there are still a few problems.
warning: 4 functions have mismatched data
It would be great to have this working better. This tool is super handy.
Currently, running with --doctests
I get some error output at the end, and it seems like the files that contain doctests get included twice in the coverage output:
error: src/block.rs: No such file or directory
warning: The file 'src/block.rs' isn't covered.
error: src/block/filter.rs: No such file or directory
warning: The file 'src/block/filter.rs' isn't covered.
error: src/block/iter.rs: No such file or directory
warning: The file 'src/block/iter.rs' isn't covered.
error: src/network.rs: No such file or directory
warning: The file 'src/network.rs' isn't covered.
error: src/protocol/addrmgr.rs: No such file or directory
warning: The file 'src/protocol/addrmgr.rs' isn't covered.
error: src/time.rs: No such file or directory
warning: The file 'src/time.rs' isn't covered.
@cloudhead Is there an example that can reproduce the error you encountered?
Yeah, though after reproducing this I realize some of these errors come from the --html
flag, and on its own, --doctests
doesn't output these errors, only in combination with --html
.
The repo is here: https://github.com/cloudhead/nakamoto
I ran: cargo llvm-cov --all --doctests --html
The issue with coverage output for files included twice appears with only --doctests
though. Basically the paths are not matched, eg.:
net/poll/src/time.rs 45 22 51.11% 14 5 64.29% 78 30 61.54% 0 0 -
src/time.rs 22 2 90.91% 4 0 100.00% 89 0 100.00% 0 0 -
These are actually the same file, but I'm guessing the second one is from the doctest run, while the first is from the unit test run, I'm not sure.
Note that there is no src/time.rs
, that path doesn't exist.
Thanks. I'll take a look.
#40 fixed all of the errors and many of the warnings that happen in cloudhead/nakamoto.
Before #40:
warning: 5 functions have mismatched data
error: src/block.rs: No such file or directory
warning: The file 'src/block.rs' isn't covered.
error: src/block/filter.rs: No such file or directory
warning: The file 'src/block/filter.rs' isn't covered.
error: src/block/iter.rs: No such file or directory
warning: The file 'src/block/iter.rs' isn't covered.
error: src/network.rs: No such file or directory
warning: The file 'src/network.rs' isn't covered.
error: src/protocol/addrmgr.rs: No such file or directory
warning: The file 'src/protocol/addrmgr.rs' isn't covered.
error: src/time.rs: No such file or directory
warning: The file 'src/time.rs' isn't covered.
warning: 5 functions have mismatched data
After #40:
warning: 5 functions have mismatched data
warning: 5 functions have mismatched data
The remaining warnings are probably the column offset errors mentioned in https://github.com/rust-lang/rust/issues/79417#issuecomment-756418399, and it's not clear whether we can fix them on our side at this time.
Awesome, thanks a bunch!
I am not sure whether I have missed any of the prerequisites, but I have tried to use the --doctests
option in the rust-numpy
crate, c.f. https://github.com/PyO3/rust-numpy/pull/286, but it appears that while the doctests are executed, they are not included in the coverage computation (whereas both unit tests and integration tests work). There are no error messages and no warnings besides the initial warning that the feature is unstable. I can also reproduce this locally, e.g. if I run cargo llvm-cov --doc --html
I get an empty report.
Can anybody advise if am missing some dependency or step? Thank you!
My understanding is that the coverage is broken due to a bug of rustc's --remap-path-prefix (#140, #122). Could you try #141?
Could you try https://github.com/taiki-e/cargo-llvm-cov/pull/141?
I installed cargo-llvm-cov using
> cargo install --force --git https://github.com/taiki-e/cargo-llvm-cov.git --branch disable-remap-path-prefix cargo-llvm-cov
but did not observe any changes in the output. I guess I need to wait for #122 and therefore https://github.com/rust-lang/rust/pull/92648 to land?
I don't know if this is related to this issue here but adding the below lines to a test will give me warning: 4 functions have mismatched data
.
let candy_color = [
String::from("#ea3223"),
String::from("#377d22"),
String::from("#fffd54"),
String::from("#ea3df7"),
String::from("#74fbfd"),
String::from("#ee776d"),
String::from("#8cf57b"),
String::from("#fffb7f"),
String::from("#6974f6"),
String::from("#ee82f8"),
String::from("#8dfafd"),
];
assert!(candy_color.contains(&color2hex(&Colors::Candy, &options)));
These lines are within a *_test.rs
file inside the test folder and inside a #[cfg(test)]
and #[test]
block.
They test the output of a function that randomizes it's output.
And I'm running the test without --doctests
via cargo llvm-cov --html
on stable
@dominikwilkowski I believe that's also a rustc bug. (maybe related to code generated by #[test]
attr)