Consider support for llvm profile compiler runtime library

[eldruin]

The approach described here uses llvm profile compiler runtime library, which has several benefits. Most interestingly, it seems to me like the coverage results are more accurate and can even include branch coverage. See for yourself:

• rusttls (llvm profile)
• nom (kcov)

However, the scripts are not that trivial and letting them duplicate through rust crates would not be very good for the ecosystem. Is there any intention to integrate this method into cargo-travis?

[roblabla]

Thanks for this, I wasn’t aware of lcov, it’s super interesting. Here’s a bit of a braindump of the various things I’m thinking about.

I’d love to support better methods of gathering coverage, but at the same time I worry changing coverage tool might break some people’s builds - especially since, given it removes landing pads, I suspect it breaks panics? Maybe with a flag, we could chose a coverage tool. The default would remain kcov for backward compat.

Are any of the flags used in that blog nightly-only? The post doesn’t mention it, but I thought playing with optimisation passes was unstable. Maybe I’m remembering wrong.

For this method, the main “complicated” part is generating the rustc_wrapper script. We need to get the name of the crates we want to integrate coverage into (making sure to avoid building coverage into build scripts and whatnot). Hopefully, I’ll be able to automate this by looking at the build targets by parsing cargo metadata.

Could this work on windows? Does llvm profile library and lcov work there? If so, we’ll want to think about ways to generate a rustc wrapper that avoids bash. Probably, write it in rust, and pass it the crate names via an env variable.

I’d also like to minimize non rust dependencies. Depending on ruby and a gem to upload the lcov data to coveralls is meh. I’d want to rewrite that in rust to keep installation of cargo-travis easy and self-contained.

[eldruin]

I'm very happy to hear about your interest.

• I also think the best way to support lcov in cargo-travis would be an additional opt-in parameter. I agree that more breakage is the last thing we need. Any syntax of your choice would be good.
• About the ruby gem, I'd be happy to write a similar multiplatform tool in Rust if we move forward with this.

About nightly-only or not and whether this would work on Windows I have no idea. Maybe we can involve the original author here. Hi @ctz!

[roblabla]

So I did a quick check, and the LLVM binary builds for windows include clang_rt.profile-x86_64.lib (full path is lib/clang/7.0.0/lib/windows/clang_rt.profile-x86_64.lib). So I guess it works on windows, which is super good news \o/. Further supporting this is a blog post about using clang code coverage on windows: https://marco-c.github.io/2018/01/09/code-coverage-with-clang-on-windows.html.

Looking at the binary release of LLVM, llvm-cov seems to be missing though! 😱. Thankfully, according to the above blog post, we can use grcov, a rust-based alternative used by firefox and maintained by mozilla. It has builds for Windows, Mac and Linux. Additionally, it can upload straight to coveralls, so we can avoid the ruby gem.

As an aside, while investigating this, I discovered https://github.com/kennytm/cov, which might be able to suit your needs in the meantime?