cargo-mutants icon indicating copy to clipboard operation
cargo-mutants copied to clipboard

Add --test-tool=nextest

Open sourcefrog opened this issue 1 year ago • 0 comments

Fixes #85

  • [x] Update manual
  • [x] Exercise from cargo-mutants' own CI
  • [x] Integration test

Performance on cargo-mutants

I tested this on cargo-mutants itself locally and in CI, and it works well. However, interestingly, it does currently seem to be significantly slower in sharded full-tree tests than is cargo test (about 8m vs >13m). (Example CI run.)

It's also somewhat slower, although not as much, locally:

; c r --release mutants --shard 1/10 --test-tool=nextest
   Compiling cargo-mutants v23.12.2 (/home/mbp/src/mutants)
    Finished release [optimized] target(s) in 4.49s
     Running `target/release/cargo-mutants mutants --shard 1/10 --test-tool=nextest`
Found 56 mutants to test
ok       Unmutated baseline in 13.9s build + 22.3s test
Auto-set test timeout to 1m 51s
56 mutants tested in 4m 22s: 40 caught, 16 unviable


; c r --release mutants --shard 1/10
    Finished release [optimized] target(s) in 0.08s
     Running `target/release/cargo-mutants mutants --shard 1/10`
Found 56 mutants to test
ok       Unmutated baseline in 13.8s build + 21.7s test
Auto-set test timeout to 1m 48s
56 mutants tested in 3m 46s: 40 caught, 16 unviable

Although it's probably better at stopping early, it may be paying a higher per-test overhead or maybe making less use of parallelism?

Performance on stackslib

I also tested this on a 1% sample of stackslib with

; c r --release mutants --shard 1/100 -d ~/src/stacks-core/ --test-tool nextest
    Finished release [optimized] target(s) in 0.18s
     Running `target/release/cargo-mutants mutants --shard 1/100 -d /home/mbp/src/stacks-core/ --test-tool nextest`
Found 112 mutants to test
ok       Unmutated baseline in 116.1s build + 244.5s test
Auto-set test timeout to 20m 22s
MISSED   src/burnchains/bitcoin/spv.rs:1150:55: replace > with == in SpvClient::get_target in 34.1s build + 256.1s test
MISSED   src/chainstate/burn/db/sortdb.rs:2893:25: replace == with != in SortitionDB::is_db_version_supported_in_epoch in 25.8s build + 283.8s test
3358.435171577s  INFO timeout after 1222.4s, terminating child process...
TIMEOUT  src/net/connection.rs:1334:9: replace NetworkConnection<P>::send_data -> Result<usize, net_error> with Ok(1) in 34.8s build + 1222.4s test
MISSED   src/chainstate/coordinator/comm.rs:105:9: replace SignalBools::receive_signal -> u8 with 0 in 38.0s build + 278.8s test
MISSED   src/main.rs:957:16: replace == with != in main in 24.3s build + 258.5s test
MISSED   src/net/p2p.rs:4235:9: replace PeerNetwork::need_block_or_microblock_stream -> Result<bool, net_error> with Ok(true) in 28.7s build + 267.8s test
MISSED   src/net/p2p.rs:1280:9: replace PeerNetwork::num_peers -> usize with 1 in 26.8s build + 311.8s test
test     src/clarity_vm/database/mod.rs:174:9: replace <impl HeadersDB for ChainstateTx<'a>>::get_burnchain_tokens_spent_for_block -> Option<u128> with Some(1) ... 233.0s
MISSED   src/net/http.rs:1730:9: replace HttpRequestType::get_proof_query -> bool with true in 33.5s build + 291.0s test
MISSED   src/util_lib/db.rs:656:20: replace > with == in tx_busy_handler in 35.5s build + 290.3s test
test     src/chainstate/stacks/db/unconfirmed.rs:443:9: replace UnconfirmedState::num_mined_txs -> usize with 0 s
└                PASS [  28.317s]                     blockstack-core clarity_vm::tests::costs::epoch_21_test_alt
47/112 mutants tested, 8 MISSED, 1 timeout, 24 caught, 14 unviable, 1h 49m 40s elapsed, about 2h 32m remaining

It looks like some of those tests are still pretty slow, but it is making progress.

sourcefrog avatar Jan 02 '24 23:01 sourcefrog