Test failures under tarpaulin (fine outside of it)

[mathstuf]

There are some test failures in a project I work on only under tarpaulin. We've been ignoring them for now, but I'd like to get to the bottom of it at some point. The reformat tests on the project fail by timing out on our "look for the expected state" loop (each test launches a background thread to do mocking). These tests are "special" in that they wait on a spawned process for its output…except that all tests do that when they perform their git operations.

Repo: https://gitlab.kitware.com/utils/ghostflow-director Logs: https://gitlab.kitware.com/utils/ghostflow-director/-/jobs/4982784

[xd009642]

Hmm it's probably related to ptrace then, I know there is a lot of room to improve the handling of spawned processes in tarpaulin. I'll try and have a look over the weekend if I have time

[BrandonDyer64]

I've also had this issue. When removing the test that was failing only under tarpaulin it started giving me this error:

running 6 tests
.Error: "Failed to get test coverage! Error: Failed to run tests: A segfault occurred while executing tests"
##[error]The process '/usr/share/rust/.cargo/bin/cargo' failed with exit code 1

The tests all pass with cargo test but fail with cargo tarpaulin

[dlaehnemann]

It seems like we have a similar error to @BrandonDyer64 over at rust-bio that only occurs when running cargo tarpaulin and not when running regular cargo test:

thread 'main' panicked at 'already borrowed: BorrowMutError', src/tools/cargo/src/cargo/util/config/mod.rs:307:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Oct 19 20:04:10.904 ERROR cargo_tarpaulin: Failed to compile tests! Error: unresolved import `crate::arch::x86_64::_mm_movemask_pi8`

[xd009642]

@dlaehnemann do you have a .cargo/config per chance? Just wondering if there could be some extra rustflags or something at play for native settings? I know simd-json experiences build issues because of that unless they set the RUSTFLAGS env var appropriately in their CI job. Tarpaulin doesn't yet pull them in automatically although there's a WIP PR to do that.

[dlaehnemann]

I'm not aware of any .cargo/config.toml files anywhere in the project's hierarchy. I would think the only places where any RUSTFLAGS could come in would be:

• the general Cargo.toml
• the build.rs that Cargo.toml references
• maybe the rather new .rustfmt.toml

Otherwise, the only thing I could think of is that we are doing something "wrong" in our GitHub Actions setup of tarpaulin: https://github.com/rust-bio/rust-bio/blob/6c5a6c20d3f36642929121a96e717a05e97047fb/.github/workflows/rust.yml#L85-L106

And finally, I have now started a branch with fully verbose tarpaulin and RUST_BACKTRACE=full. It doesn't get me any further in my understanding, but maybe you or someone else can make more of it: https://github.com/rust-bio/rust-bio/pull/389/checks?check_run_id=1280007356#step:5:59

[dlaehnemann]

@xMAC94x suggested in another issue where I mentioned this that the error could be due to this rust-lang bug. So this might as well be upstream.

[djeedai]

I'm having suddenly the same issue, see https://github.com/djeedai/weldr/runs/1397092538#step:8:24 The project is very small, and tests are running fine outside tarpaulin (cargo test). This is a 100% repro. I tried some workarounds from https://github.com/rust-lang/rust/issues/25289 like installing gcc-multilib or running cargo clean, without much success. Any idea what could be causing that issue?

Callstack for reference:

   Compiling weldr v0.1.0 (/home/runner/work/weldr/weldr)
error: Broken pipe (os error 32)
warning: build failed, waiting for other jobs to finish...
thread 'main' panicked at 'already borrowed: BorrowMutError', src/tools/cargo/src/cargo/util/config/mod.rs:302:20
stack backtrace:
   0: rust_begin_unwind
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:475
   1: core::panicking::panic_fmt
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/panicking.rs:85
   2: core::option::expect_none_failed
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/option.rs:1221
   3: cargo::core::compiler::job_queue::DrainState::drain_the_queue
   4: std::panic::catch_unwind
   5: crossbeam_utils::thread::scope
   6: cargo::core::compiler::job_queue::JobQueue::execute
   7: cargo::core::compiler::context::Context::compile
   8: cargo::ops::cargo_compile::compile_ws
   9: cargo::ops::cargo_compile::compile
  10: cargo::ops::cargo_test::run_tests
  11: cargo::commands::test::exec
  12: cargo::cli::main
  13: cargo::main

[djeedai]

I can also confirm that this is likely a regression from 0.16.0 -> 0.17.0 Here's the passing CI when forcing 0.16.0 : https://github.com/djeedai/weldr/runs/1399701115 The failures started very precisely when CI picked up 0.17.0 instead of 0.16.0, as I was previously not requesting any specific version. I am now working around by staying on 0.16.0, which I don't think will be a problem for me.

[svenstaro]

I found that I get no

thread 'main' panicked at 'already borrowed: BorrowMutError', src/tools/cargo/src/cargo/util/config/mod.rs:302:20

when running cargo clean before cargo tarpaulin.

[dlaehnemann]

Thanks for the idea, but sadly the cargo clean did not do the job for us. And for us, the error already appeared in 0.16.0.

[bytesnake]

I ran into the same issue when using tarpaulin for the whole workspace in rust-ml/linfa, this runs fine

cargo +nightly tarpaulin --all-features -v --run-types Tests Doctests

but passing --workspace fails with thread 'main' panicked at 'already borrowed: BorrowMutError', src/tools/cargo/src/cargo/util/config/mod.rs:307:20

[msrd0]

I also just ran into this when dependabot updated the bytes dependency to a newer but incompatible version: https://github.com/msrd0/gotham_formdata/pull/18/checks?check_run_id=1464051940

error: Broken pipe (os error 32)
warning: build failed, waiting for other jobs to finish...
thread 'main' panicked at 'already borrowed: BorrowMutError', src/tools/cargo/src/cargo/util/config/mod.rs:307:20
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:483
   1: core::panicking::panic_fmt
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/panicking.rs:85
   2: core::option::expect_none_failed
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/option.rs:1234
   3: cargo::core::compiler::job_queue::DrainState::drain_the_queue
   4: std::panic::catch_unwind
   5: crossbeam_utils::thread::scope
   6: cargo::core::compiler::job_queue::JobQueue::execute
   7: cargo::core::compiler::context::Context::compile
   8: cargo::ops::cargo_compile::compile_ws
   9: cargo::ops::cargo_compile::compile
  10: cargo::ops::cargo_test::compile_tests
  11: cargo::ops::cargo_test::run_tests
  12: cargo::commands::test::exec
  13: cargo::cli::main
  14: cargo::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Error: "Failed to compile tests! Error: mismatched types"

Running cargo test produced the expected error message, so this seems to be related to tarpaulin. Note that the cache would contain bytes 0.5, not bytes 0.6, but that shouldn't cause a panic.

[xd009642]

For anyone still having issues if you want to try with --engine llvm or try again on latest tarpaulin there have been a lot of changes in last 3 years. Including the llvm coverage support which was a big chunk of my effort on tarpaulin. I'm going to close this as it's pretty old but if you still have issues feel free to open new issues with behaviour on latest tarpaulin

[djeedai]

Yes I've switched to LLVM a few months ago with djeedai/bevy_hanabi#82 and things look good now. Thanks!