tarpaulin icon indicating copy to clipboard operation
tarpaulin copied to clipboard

LLVM coverage instrumentation

Open xd009642 opened this issue 3 years ago • 31 comments

So when llvm coverage instrumentation is stable on rust we can look at removing the ptrace based coverage completely and instead relying on llvm and rustc to help. This has a few advantages

  1. None of the weirdness of ptrace
  2. More accurate results
  3. First class support in the compiler - less chance of behaviour changing in nightly
  4. Linker issues, large target folders and slowness should be greatly mitigated
  5. Easier route to get branch/condition and other coverage metrics by coupling llvm_lines and source analysis

In this instance tarpaulin should make the whole experience still nicer because:

  1. Handles calling cargo and right flags
  2. Doc tests coverage without needing to remember the nightly flags
  3. results collation/report generation
  4. tagging unused generics that won't be instrumented

OSX TODOs:

  • [x] doc_coverage::doc_test_coverage
  • [x] doc_coverage::doc_test_env
  • [x] doc_coverage::doc_test_panics
  • [x] doc_coverage::doc_test_panics_workspace
  • [x] failures::error_if_test_fails
  • [x] follow_exes_down
  • [x] handle_forks
  • [x] issue_966_follow_exec
  • [x] kill_used_in_test
  • [x] match_expr_coverage
  • [x] rustflags_handling
  • [x] test_types::only_test_coverage

Windows TODOs

  • [x] failures::error_if_test_fails
  • [x] follow_exes_down
  • [x] issue_966_follow_exec
  • [x] match_expr_coverage
  • [x] rustflags_handling
  • [x] picking_up_shared_objects
  • [x] doc_coverage::doc_test_panics
  • [x] doc_coverage::doc_test_panics_workspace

Linux TODOs

  • [x] doc_coverage::doc_test_panics
  • [x] failures::error_if_test_fails
  • [x] handle_forks
  • [x] follow_exes_down
  • [x] issue_966_follow_exec
  • [x] match_expr_coverage
  • [x] rustflags_handling
  • [x] test_types::only_test_coverage

General TODO

  • [x] Coverage behaviour with loops

xd009642 avatar Aug 29 '20 10:08 xd009642

https://hammer-vlsi.readthedocs.io/en/stable/CoverageMappingFormat.html some docs and the rust issue I keep having to find again https://github.com/rust-lang/rust/issues/34701

xd009642 avatar Oct 13 '20 20:10 xd009642

Is there already a branch where we can try this?

caemor avatar Nov 27 '20 10:11 caemor

Not yet, given my current work schedule I'm hoping to have something early January

xd009642 avatar Nov 27 '20 10:11 xd009642

Work schedule and covid restrictions came back to bite me I'm likely to be abroad away from my home until march or april until I can return. I plan on starting the work to make it so tarpaulin compiles on other OSs even if it doesn't do anything and then bring in llvm coverage

resources found: https://opensource.apple.com/source/clang/clang-700.0.72/src/docs/CoverageMappingFormat.rst.auto.html

xd009642 avatar Feb 17 '21 13:02 xd009642

So I've merged the first PR, didn't want to be constantly building up a giant PR that becomes hard to merge :sweat_smile:. This one simply adds a future option (not visible to the user) to set the tracing engine. It has 3 options Auto, Ptrace and llvm (and with #675 I hope to add probe-rs). So currently this is hard set to Ptrace, but if it's set to Auto or llvm and the compiler version is compatible it will build the test binaries with the llvm-cov instrumentation included. If set to llvm and the compiler isn't compatible it prints an error log warning the user the compiler doesn't support it and falls back to ptrace.

There are some issues with doctests marked as should_panic and the llvm coverage but everything else works fine and generates the profdata files for the tests and passes as expected.

Looking at what I can see of the profdata it looks like I have all the info to parse it with the binary and I'd ideally like to do that from the get-go to avoid making the user install the llvm-tools component and using llvm-profdata. Another benefit is it also gives me ability to access the information I need to derive other coverage metrics than line coverage which would be a big future win. I haven't found a source for the format other than the llvm source code which isn't quite a joy to read...

xd009642 avatar Feb 18 '21 19:02 xd009642

💖 Another Sponsor here rooting for #549!

Go @xd009642!

mikeumus avatar Apr 24 '21 21:04 mikeumus

So as another update I've just merged https://github.com/xd009642/tarpaulin/pull/786 which allows for selecting the coverage engine to use and actually runs the llvm-cov instrumented binaries and just prints out the path to the test executable and the profraw files it generates for each test. It doesn't yet collect the metrics from within those files. I've also made the flag to select the engine backend hidden so people who want to experiment can but for normal users they won't be able to access incomplete features.

For my aim to parse the formats in Rust to avoid installing extra tools, I've been working on https://github.com/xd009642/llvm-profparser I can now parse and merge the profraw files and the consistency tests show the implementation matches that of the llvm tools using their own test vectors. The next step is getting the counter to source mapping which is compiled into the test binary to turn those coverage counters into actual coverage stats. Once that's done I just need to tidy up the error handling as there's a few unwraps/todos in the code still and it should be ready to integrate.

xd009642 avatar Jul 08 '21 08:07 xd009642

Another small update, the llvm-profparser parsing of the ELF sections has progressed and I'm close. Just failing to exactly see how the two parsed formats interlink as the counters seem different. I expect this is to do with the addition/subtraction expressions being like span deltas for coverage and I might be able to reduce those expressions down so the counter counts match. We'll see, I want to check the LLVM code/docs to double check the correctness/implementation of that.

So failing the motivation to take that final jump tonight I published an alpha verion of my llvm-profparser crate and integrated it into tarpaulin. It panics due to a todo!() but we can see the parsed formats printed out just before that!

 tarpaulin --engine llvm -r tests/data/simple_project/`
Nov 09 22:12:22.243  INFO cargo_tarpaulin::config: Creating config
Nov 09 22:12:22.527  INFO cargo_tarpaulin: Running Tarpaulin
Nov 09 22:12:22.527  INFO cargo_tarpaulin: Building project
Nov 09 22:12:22.527  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling simple_project v0.1.0 (/home/daniel/personal/tarpaulin/tests/data/simple_project)
    Finished test [unoptimized + debuginfo] target(s) in 0.57s
Nov 09 22:12:23.219  INFO cargo_tarpaulin::process_handling: running /home/daniel/personal/tarpaulin/tests/data/simple_project/target/debug/deps/simple_project-e5714f4a5f742fbe

running 1 test
test tests::bad_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Nov 09 22:12:24.558  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/simple_project-e5714f4a5f742fbe
Nov 09 22:12:24.558  INFO cargo_tarpaulin::statemachine::instrumented: Generated: simple_project-e5714f4a5f742fbe_19330.profraw
Profile:
InstrumentationProfile { version: Some(7), has_csir: false, is_ir: false, is_entry_first: false, records: [NamedInstrProfRecord { name: Some("_RNCNvNtCsdQnHFJaUqdt_14simple_project5tests8bad_test0B5_"), hash: Some(17287578574987469320), record: InstrProfRecord { counts: [1, 0], data: None } }, NamedInstrProfRecord { name: Some("_RNvNtCsdQnHFJaUqdt_14simple_project6unused5hello"), hash: Some(11134845332641192253), record: InstrProfRecord { counts: [0, 0], data: None } }, NamedInstrProfRecord { name: Some("_RNvCsdQnHFJaUqdt_14simple_project15branch_test_one"), hash: Some(12048548525588240215), record: InstrProfRecord { counts: [1, 0, 0], data: None } }, NamedInstrProfRecord { name: Some("_RNvCsdQnHFJaUqdt_14simple_project4main"), hash: Some(4823579800516870695), record: InstrProfRecord { counts: [1, 0], data: None } }, NamedInstrProfRecord { name: Some("_RNvNtCsdQnHFJaUqdt_14simple_project5testss_8bad_test"), hash: Some(3214838615682077355), record: InstrProfRecord { counts: [1, 0], data: None } }], symtab: Symtab { names: {1882277207708802556: "_RNvCsdQnHFJaUqdt_14simple_project4main", 11053755068919242637: "_RNvNtCsdQnHFJaUqdt_14simple_project5testss_8bad_test", 13551117777733895944: "_RNCNvNtCsdQnHFJaUqdt_14simple_project5tests8bad_test0B5_", 14510898034703653526: "_RNvNtCsdQnHFJaUqdt_14simple_project6unused5hello", 15771523483848882675: "_RNvCsdQnHFJaUqdt_14simple_project15branch_test_one"} } }
Mappings:
[CoverageMappingInfo { cov_map: ["src/lib.rs", "src/unused.rs", "src/lib.rs", "src/lib.rs"], cov_fun: [FunctionRecordV3 { header: FunctionRecordHeader { name_hash: 13551117777733895944, data_len: 9, func_hash: 17287578574987469320, filenames_ref: 6184173430748275366 }, regions: [CounterMappingRegion { kind: Code, count: Counter { kind: ProfileInstrumentation, id: 0 }, file_id: 0, expanded_file_id: 0, line_start: 20, column_start: 5, line_end: 20, column_end: 12 }] }, FunctionRecordV3 { header: FunctionRecordHeader { name_hash: 14510898034703653526, data_len: 9, func_hash: 11134845332641192253, filenames_ref: 9320222428158223143 }, regions: [CounterMappingRegion { kind: Code, count: Counter { kind: ProfileInstrumentation, id: 0 }, file_id: 0, expanded_file_id: 0, line_start: 4, column_start: 1, line_end: 7, column_end: 2 }] }, FunctionRecordV3 { header: FunctionRecordHeader { name_hash: 15771523483848882675, data_len: 28, func_hash: 12048548525588240215, filenames_ref: 6184173430748275366 }, regions: [CounterMappingRegion { kind: Code, count: Counter { kind: ProfileInstrumentation, id: 0 }, file_id: 0, expanded_file_id: 0, line_start: 4, column_start: 1, line_end: 5, column_end: 13 }, CounterMappingRegion { kind: Code, count: Counter { kind: ProfileInstrumentation, id: 1 }, file_id: 0, expanded_file_id: 0, line_start: 6, column_start: 9, line_end: 6, column_end: 11 }, CounterMappingRegion { kind: Code, count: Counter { kind: SubtractionExpr, id: 0 }, file_id: 0, expanded_file_id: 0, line_start: 8, column_start: 9, line_end: 8, column_end: 10 }, CounterMappingRegion { kind: Code, count: Counter { kind: AdditionExpr, id: 1 }, file_id: 0, expanded_file_id: 0, line_start: 10, column_start: 1, line_end: 10, column_end: 2 }] }, FunctionRecordV3 { header: FunctionRecordHeader { name_hash: 1882277207708802556, data_len: 9, func_hash: 4823579800516870695, filenames_ref: 6184173430748275366 }, regions: [CounterMappingRegion { kind: Code, count: Counter { kind: ProfileInstrumentation, id: 0 }, file_id: 0, expanded_file_id: 0, line_start: 1, column_start: 0, line_end: 1, column_end: 1 }] }, FunctionRecordV3 { header: FunctionRecordHeader { name_hash: 11053755068919242637, data_len: 9, func_hash: 3214838615682077355, filenames_ref: 6184173430748275366 }, regions: [CounterMappingRegion { kind: Code, count: Counter { kind: ProfileInstrumentation, id: 0 }, file_id: 0, expanded_file_id: 0, line_start: 21, column_start: 5, line_end: 23, column_end: 6 }] }], prof_names: ["_RNCNvNtCsdQnHFJaUqdt_14simple_project5tests8bad_test0B5_", "_RNvNtCsdQnHFJaUqdt_14simple_project6unused5hello", "_RNvCsdQnHFJaUqdt_14simple_project15branch_test_one\u{1}_RNvCsdQnHFJaUqdt_14simple_project4main", "_RNvNtCsdQnHFJaUqdt_14simple_project5testss_8bad_test"], prof_counts: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], prof_data: [ProfileData { name_md5: 13551117777733895944, structural_hash: 17287578574987469320, counters_len: 2 }, ProfileData { name_md5: 14510898034703653526, structural_hash: 11134845332641192253, counters_len: 2 }, ProfileData { name_md5: 15771523483848882675, structural_hash: 12048548525588240215, counters_len: 3 }, ProfileData { name_md5: 1882277207708802556, structural_hash: 4823579800516870695, counters_len: 2 }, ProfileData { name_md5: 11053755068919242637, structural_hash: 3214838615682077355, counters_len: 2 }] }]
thread 'main' panicked at 'not yet implemented', /home/daniel/.cargo/registry/src/github.com-1ecc6299db9ec823/llvm_profparser-0.1.0-alpha1/src/coverage/coverage_mapping.rs:98:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

EDIT: Also it's in the intuitively named feat/llvm-profparser-integration branch

xd009642 avatar Nov 09 '21 22:11 xd009642

Also, after some work with a very-early adopter interested in windows support there's now a commit pushed that simplifies that and should hopefully mean windows users can get to the same todo! panic linux users do without hitting any issues beforehand :eyes:

xd009642 avatar Nov 10 '21 08:11 xd009642

I see that rustc coverage instrumentation was stabilized in the 1.60 release today. Is it fair to say that what is included is sufficient to switch the underlying coverage support and implement this feature?

nu11ptr avatar Apr 07 '22 18:04 nu11ptr

The short answer: no.

The long answer: needing to install cargo-binutils separately and rely on that being stable or finding the correct llvm-tools in the rust install (handling distro installs or rustup) felt like a pretty poor solution to me (partially because I already know there will be a large number of issues where the answer is "run rustup component add llvm-tools-preview && cargo install cargo-binutils). Also for some things I was planning like embedded coverage in future they wouldn't necessarily work at all.

So I went straight for the solution of implementing the profdata/profraw parsing in https://github.com/xd009642/llvm-profparser which currently works with llvm 11, 12, 13 and 14 formats (passing all their test suite). I also correctly seem to parse the ELF headers LLVM adds to the binary so the missing piece is mapping between the profdata and the binary source locations and then tarpaulin will work with llvm coverage on any installed compiler version that has the llvm coverage without need for any additional tools. Parsing the profraw and profdata was actually reasonably quick it's just finding time for the last bits as I don' have much personal time right now

xd009642 avatar Apr 07 '22 18:04 xd009642

Well after a bunch of work over last week I now have the very initial version working all the way through. Anyone who wants to test and report any issues would be appreciated. I'll be running it through some tests and polishing more before release but it's the home stretch.

The branch to install from for testing is feat/llvm-profparser-integration and as mentioned above there's no need for llvm-tools-preview or cargo-binutils just need a recent enough rust compiler and cargo :smile:

xd009642 avatar Jun 05 '22 18:06 xd009642

Some more movement on this, I've fixed an issue with it on windows. Doing some more tests make sure everything passes CI then will look at moving off alpha versions and doing an initial release with using llvm coverage as a non-default option. And then making default when I've had more a chance to look at the output at a lower level.

xd009642 avatar Jun 26 '22 08:06 xd009642

@xd009642 I think I've found a bug:

$ RUST_BACKTRACE=1 ../tarpaulin/target/debug/cargo-tarpaulin tarpaulin --engine llvm
⋮

Jun 26 12:41:20.803  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/cargo_rdme-8cd2fbe1c7e1b41d
Jun 26 12:41:20.803  INFO cargo_tarpaulin::statemachine::instrumented: Generated: cargo_rdme-8cd2fbe1c7e1b41d_1033777.profraw
thread 'main' panicked at 'attempt to subtract with overflow', /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/arith.rs:240:1
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::panicking::panic
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:48:5
   3: <u64 as core::ops::arith::Sub>::sub
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/arith.rs:233:45
   4: <&u64 as core::ops::arith::Sub<&u64>>::sub
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/internal_macros.rs:80:17
   5: llvm_profparser::coverage::coverage_mapping::CoverageMapping::generate_report
             at /home/orium/.cargo/registry/src/github.com-1ecc6299db9ec823/llvm_profparser-0.1.1-alpha1/src/coverage/coverage_mapping.rs:206:55
   6: <cargo_tarpaulin::statemachine::instrumented::LlvmInstrumentedData as cargo_tarpaulin::statemachine::StateData>::wait
             at /home/orium/programming/projects/tarpaulin/src/statemachine/instrumented.rs:99:34
   7: <alloc::boxed::Box<dyn cargo_tarpaulin::statemachine::StateData> as cargo_tarpaulin::statemachine::StateData>::wait
             at /home/orium/programming/projects/tarpaulin/src/statemachine/mod.rs:156:9
   8: cargo_tarpaulin::statemachine::TestState::step
             at /home/orium/programming/projects/tarpaulin/src/statemachine/mod.rs:204:34
   9: cargo_tarpaulin::process_handling::collect_coverage
             at /home/orium/programming/projects/tarpaulin/src/process_handling/mod.rs:147:21
  10: cargo_tarpaulin::process_handling::get_test_coverage
             at /home/orium/programming/projects/tarpaulin/src/process_handling/mod.rs:74:15
  11: cargo_tarpaulin::launch_tarpaulin
             at /home/orium/programming/projects/tarpaulin/src/lib.rs:221:28
  12: cargo_tarpaulin::trace
             at /home/orium/programming/projects/tarpaulin/src/lib.rs:92:15
  13: cargo_tarpaulin::collect_tracemap
             at /home/orium/programming/projects/tarpaulin/src/lib.rs:165:24
  14: cargo_tarpaulin::run
             at /home/orium/programming/projects/tarpaulin/src/lib.rs:160:20
  15: cargo_tarpaulin::main
             at /home/orium/programming/projects/tarpaulin/src/main.rs:67:5
  16: core::ops::function::FnOnce::call_once
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I minimized the issue to this: https://github.com/orium/cargo-rdme/blob/bug-tarpaulin-llvm-profparser/src/main.rs#L6-L12

orium avatar Jun 26 '22 11:06 orium

Awesome, thanks @orium. I've done a tweak on my local copy and cargo-rdme is not getting beyond that point so hopefully it's fixed. I've also spotted out a windows issue in report generation so I'll update when I've got an updated version with both fixes in :+1:

xd009642 avatar Jun 26 '22 13:06 xd009642

No worries :+1:. I can confirm the bug is gone. Later today I'll run all tests and see if any issue pops up. Something that I also want to test is --follow-exec. Is --follow-exec expected to work with the new llvm instrumentation implementation?

orium avatar Jun 26 '22 13:06 orium

So provided the RUSTFLAGS that the test is built and ran with aren't overridden by the test to build and run a binary you want the exec to follow it should "just work" in theory. Just because the followed-exec binaries will generate .profraw files that I pick up by crawling the project tree after tests are ran to find the generated ones.

This probably makes --follow-exec a default unless I added some name matching on the tests vs profraws a few months ago and forgot. Although as mention in previous paragraph, the default is override-able based on how the followed binaries are built

xd009642 avatar Jun 26 '22 14:06 xd009642

So windows is now working for me! I'm going to look at enabling all the tests in CI and see what falls out. But getting close :crossed_fingers:

xd009642 avatar Jun 27 '22 19:06 xd009642

Failing integration tests on windows 6/51 (for my own notes):

  • [x] cargo_run_coverage
  • [x] access_env_var
  • [x] examples_coverage
  • [ ] failures::error_if_test_fails
  • [ ] follow_exes_down
  • [x] issue_610
  • [x] handle_forks
  • [ ] issue_966_follow_exec
  • [ ] match_expr_coverage
  • [x] rustflags_config_coverage
  • [ ] rustflags_handling
  • [x] config_file_coverage
  • [ ] picking_up_shared_objects
  • [x] doc_coverage::doc_test_compile_fail
  • [x] doc_coverage::doc_test_coverage
  • [x] doc_coverage::doc_test_env
  • [x] doc_coverage::doc_test_panics
  • [x] doc_coverage::doc_test_panics_workspace

xd009642 avatar Jun 27 '22 22:06 xd009642

I tested the --follow-exec in a crate I have. The integration tests run the crate's binary to check that it does the right thing (https://github.com/orium/cargo-rdme/blob/tarpaulin-engine-llvm-follow-exec/tests/tests.rs#L23). The issue seems to be that the crate's binary is not part of the test binary, since it is not linked against it, it just executes it.

This is the output from tarpaulin (commit 9fe80a59e6df384ac43499cea37d28e69b829ecb):

$ ../tarpaulin/target/release/cargo-tarpaulin tarpaulin --out Html --engine llvm --debug --follow-exec
⋮
cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/cargo_rdme-894d2ebeb81d0eda
cargo_tarpaulin::statemachine::instrumented: Generated: cargo_rdme-894d2ebeb81d0eda_3211977.profraw
cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/tests-a75acdeaf82f49f5
cargo_tarpaulin::statemachine::instrumented: Generated: tests/simple_single_marker/tests-a75acdeaf82f49f5_3211980.profraw
cargo_tarpaulin::statemachine::instrumented: Generated: tests-a75acdeaf82f49f5_3211978.profraw
⋮

Note that the profraw that contains the coverage information of the process the integration runs in https://github.com/orium/cargo-rdme/blob/tarpaulin-engine-llvm-follow-exec/tests/tests.rs#L23 is this one:

cargo_tarpaulin::statemachine::instrumented: Generated: tests/simple_single_marker/tests-a75acdeaf82f49f5_3211980.profraw

but since binary target/debug/deps/tests-a75acdeaf82f49f5 doesn't know about src/main.rs we don't get coverage of these lines: https://github.com/orium/cargo-rdme/blob/tarpaulin-engine-llvm-follow-exec/src/main.rs#L2-L4

This is the report from tarpaulin:

screenshot-20220629-230451

If you run llvm-cov in the same profraw but with the binary the integration tests execute we get the correct coverage:

$ llvm-profdata merge -sparse tests/simple_single_marker/tests-a75acdeaf82f49f5_3211980.profraw -o tests/simple_single_marker/tests-a75acdeaf82f49f5_3211980.profdata
$ llvm-cov show target/debug/cargo-rdme -format=html -instr-profile=tests/simple_single_marker/tests-a75acdeaf82f49f5_3211980.profdata --ignore-filename-regex='/home/orium/.cargo/registry' > index.html

This is the report llvm-cov generates:

screenshot-20220629-230910

orium avatar Jun 29 '22 22:06 orium

cargo tarpaulin --engine llvm --doc --debug on fugit on macOS:

Jul 01 14:03:07.850 ERROR cargo_tarpaulin: Failed to get test coverage! Error: Invalid argument (os error 22)
Jul 01 14:03:07.850  INFO cargo_tarpaulin::event_log: Serializing tarpaulin debug log to tarpaulin_2022-07-01T14:03:07-05:00.json
Error: "Failed to get test coverage! Error: Invalid argument (os error 22)"

fugit-tarpaulin-doc-test.log

Anyway I can help debug the OS error?

TDHolmes avatar Jul 01 '22 19:07 TDHolmes

It may take a bit of println debugging in the code, but a first step would be to run /Users/tdh/projects/fugit/target/doctests/src_duration_rs_461_1/rust_out outside of tarpaulin and make sure it can actually run. Because on windows and linux the equivalent file will run fine right now. But doc test coverage is a bit special and uses unstable features so there may be issues

xd009642 avatar Jul 01 '22 19:07 xd009642

All of the doc tests do seem to run (all of the ones named rust_out inside of target/doctests). I'll continue to poke around 👍🏼

TDHolmes avatar Jul 01 '22 19:07 TDHolmes

Seems to be failing in generate_tracemap:

let file = unsafe { MmapOptions::new().map(&file)? };

It's trying to open the .dSYM, but isn't that a folder? /Users/tdh/projects/fugit/target/doctests/src_duration_rs_461_1/rust_out.dSYM

TDHolmes avatar Jul 01 '22 20:07 TDHolmes

Ahhh I remember when I first worked on mac coverage a few years ago there was a separate dSYM file for debug info not a folder. So that's from an early PR adding in the binary analysis side of tarpaulin before trying to sort tracing. I'll just remove that as all it can do is marginally improve the llvm line coverage. Thanks for tracking it down :tada:

xd009642 avatar Jul 01 '22 21:07 xd009642

Happy to help! Can't wait to use LLVM coverage more

TDHolmes avatar Jul 01 '22 21:07 TDHolmes

Which LLVM compiler can be considered "compatible" though? I have LLVM installed on Mac M1, but getting a warning

ERROR cargo_tarpaulin::config: unable to utilise llvm coverage, due to compiler support. Falling back to Ptrace

Which results in an ERR still.

$ which llvm-cov
/opt/homebrew/opt/llvm/bin/llvm-cov

$ llvm-cov --version

Homebrew LLVM version 14.0.6
  Optimized build.
  Default target: arm64-apple-darwin21.5.0
  Host CPU: cyclone

UPD

I've discovered this piece:

https://github.com/xd009642/tarpaulin/blob/b97f20ac86b9727d597ef561bb9c20ff28eb8090/src/cargo.rs#L37-L40

So this works for me:

$ rustup run nightly cargo tarpaulin --engine llvm -o html

xlab avatar Jul 15 '22 13:07 xlab

Have you installed from the feature branch instead of latest tarpaulin release? And compatibility is purely on the rustc version it doesn't look at llvm versioning at all. So on feature branch it's nightly > 1.50 or stable > 1.60. I guess you've not installed using the git url and branch feat/llvm-profparser-integration

https://github.com/xd009642/tarpaulin/blob/feat/llvm-profparser-integration/src/cargo.rs#L37-L41

xd009642 avatar Jul 15 '22 13:07 xd009642

Did anybody test it with docker (macOs M1). I tried it in VM (parallels + Linux ubuntu arm64) and docker (macOs m1) and I'm getting errors. Is it even supported in the mentioned configurations? It compiles fine on macOs (m1) but doesn't seem to be working with docker/linux arm (running in macos M1)?

cargo install cargo-tarpaulin --git https://github.com/xd009642/tarpaulin --branch feat/llvm-profparser-integration

Logs:

 > [3/4] RUN rustup --version;     cargo --version;     rustc --version;     cargo install cargo-tarpaulin --git https://github.com/xd009642/tarpaulin --branch feat/llvm-profparser-integration;:                    
#7 0.190 info: This is the version for the rustup toolchain manager, not the rustc compiler.                                                                                                                          
#7 0.190 rustup 1.24.3 (ce5817a94 2021-05-31)                                                                                                                                                                         
#7 0.335 info: The currently active `rustc` version is `rustc 1.57.0 (f1edd0429 2021-11-29)`                                                                                                                          
#7 0.397 cargo 1.57.0 (b2e52d7ca 2021-10-21)                                                                                                                                                                          
#7 0.448 rustc 1.57.0 (f1edd0429 2021-11-29)
#7 0.514     Updating git repository `https://github.com/xd009642/tarpaulin`
#7 2.001   Installing cargo-tarpaulin v0.20.1 (https://github.com/xd009642/tarpaulin?branch=feat/llvm-profparser-integration#f546e78c)
#7 2.271     Updating crates.io index
#7 20.45     Updating git repository `https://github.com/xd009642/llvm-profparser`
#7 21.98  Downloading crates ...
#7 27.49   Downloaded thread_local v1.1.4
#7 27.50   Downloaded tinyvec_macros v0.1.0
....
....
#7 71.06    Compiling coveralls-api v0.5.0
#7 71.62    Compiling cargo-tarpaulin v0.20.1 (/usr/local/cargo/git/checkouts/tarpaulin-6309c049c765d204/f546e78)
#7 71.86    Compiling num_enum v0.5.7
#7 81.53    Compiling git2 v0.14.2
#7 82.97 error[E0412]: cannot find type `Persona` in this scope
#7 82.97   --> src/process_handling/linux.rs:24:26
#7 82.97    |
#7 82.97 24 | const ADDR_NO_RANDOMIZE: Persona = 0x004_0000;
#7 82.97    |                          ^^^^^^^ not found in this scope
#7 82.97 
#7 82.97 error[E0412]: cannot find type `Persona` in this scope
#7 82.97   --> src/process_handling/linux.rs:25:20
#7 82.97    |
#7 82.97 25 | const GET_PERSONA: Persona = 0xFFFF_FFFF;
#7 82.97    |                    ^^^^^^^ not found in this scope
#7 82.97 
#7 82.97 error[E0412]: cannot find type `Persona` in this scope
#7 82.97   --> src/process_handling/linux.rs:74:25
#7 82.97    |
#7 82.97 74 | fn personality(persona: Persona) -> nix::Result<c_int> {
#7 82.97    |                         ^^^^^^^ not found in this scope
#7 82.97 
#7 82.97 error[E0425]: cannot find value `INT` in this scope
#7 82.97   --> src/process_handling/breakpoint.rs:60:21
#7 82.97    |
#7 82.97 60 |         intdata |= (INT << self.shift) as i64;
#7 82.97    |                     ^^^ not found in this scope
#7 82.97 
#7 83.00 warning: unused import: `c_long`
#7 83.00   --> src/process_handling/linux.rs:10:24
#7 83.00    |
#7 83.00 10 | use nix::libc::{c_int, c_long};
#7 83.00    |                        ^^^^^^
#7 83.00    |
#7 83.00    = note: `#[warn(unused_imports)]` on by default
#7 83.00 
#7 83.67 Some errors have detailed explanations: E0412, E0425.
#7 83.67 For more information about an error, try `rustc --explain E0412`.
#7 83.68 warning: `cargo-tarpaulin` (lib) generated 1 warning
#7 83.68 error: could not compile `cargo-tarpaulin` due to 4 previous errors; 1 warning emitted
#7 83.68 warning: build failed, waiting for other jobs to finish...
#7 84.54 error: failed to compile `cargo-tarpaulin v0.20.1 (https://github.com/xd009642/tarpaulin?branch=feat/llvm-profparser-integration#f546e78c)`, intermediate artifacts can be found at `/tmp/cargo-installgEgHW0`
#7 84.54 
#7 84.54 Caused by:
#7 84.54   build failed
------
executor failed running [/bin/sh -c rustup --version;     cargo --version;     rustc --version;     cargo install cargo-tarpaulin --git https://github.com/xd009642/tarpaulin --branch feat/llvm-profparser-integration;]: exit code: 101

karolh2000 avatar Jul 28 '22 20:07 karolh2000

A version from feat/llvm-profparser-integration branch works successfully for me on Mac M1 (not in docker though, I haven't tried). Thanks, @xd009642!

rusxg avatar Sep 12 '22 06:09 rusxg