How do I calculate code coverage of a CLI tool (not a test)?

[vi]

Tarpaulin (or at least it's README) seems to be oriented towards running unit tests.

Even with --command build it thinks the tool is a test of some sort.

How do I use Tarpaulin to aggregate cover coverage of multiple runs of the instrumented binary which is started externally, not by Tarpaulin itself?

[vi]

Seems like -Zinstrument-coverage + llvm-profdata + llvm-cov covers my needs.

Maybe Tarpaulin's README should explicitly mention it's nonusage for nontests, linking articles about how to do it manually.

[xd009642]

So I'm not sure what you mean it thinks the tool is a test of some sort? Is it just because the log message says "launching test" because that is an oversight I just realised existed now, but it does just launch any binary that it works out it has to run and I'll change that log message to be more descriptive.

So would this tool be something like a server that you want to keep running and send multiple requests to? Because doing multiple runs of a CLI app where it's started for each run you can use config files and do something like:

[run_one]
command = build
args = ["command1", "--arg"]

[run_two]
command = build
args = ["command2", "--arg", "value"]

Incremental compilation will be able to properly take hold once rust 1.53 is released with some recent PRs and avoid the cleaning for each run which is admittedly an annoyance right now.

[vi]

For example, it can be used to capture code coverage of a tool that is tested interactively (manually), without a real test suite.

Or where the tool uses some Rust library, but is primarily written in something else and uses foreign test runner (which is not directly under tarpaulin's control).

I don't expect for Tarpaulin to provide the "turnkey" code coverage solution in this case (just run one command and get the coverage report), but it can still provide some lower level commands to build the instrumented executable/library without attempting to run it, plus a command to generate the report based on explicitly specified instrumentation output files.

[xd009642]

okay I've got a clearer idea of what's needed now. There is --no-run to build the binaries without running them. For ptrace coverage unfortunately tarpaulin needs to run the tests (although maybe there's room to use ptrace seize or attach to get already running processes but that goes to an even less documented part of a rather poorly documented API).

Once I've got the llvm coverage stuff fully integrated I can always add a tarpaulin subcommand to generate reports from the profraw files etc left behind from running via other means which I think would get us the rest of the way to handling your use case. That's probably a way out though so I'm glad you've found a working solution with llvm-profdata/llvm-cov etc :+1:

[xd009642]

It's a few versions old now but there's now a secret flag to build llvm instrumented binaries. Keeping it undocumented while the rest of the coverage solution isn't hooked in. But now by going cargo +nightly tarpaulin --engine llvm --no-run you can build tests with llvm instrumentation. For building a specific CLI binary you'll likely have to cargo +nightly tarpaulin --engine llvm --no-run --run-types Bins --bin $BINARY_NAME