cargo-mutants
cargo-mutants copied to clipboard
Add --test-tool=nextest
Fixes #85
- [x] Update manual
- [x] Exercise from cargo-mutants' own CI
- [x] Integration test
Performance on cargo-mutants
I tested this on cargo-mutants itself locally and in CI, and it works well. However, interestingly, it does currently seem to be significantly slower in sharded full-tree tests than is cargo test (about 8m vs >13m). (Example CI run.)
It's also somewhat slower, although not as much, locally:
; c r --release mutants --shard 1/10 --test-tool=nextest
Compiling cargo-mutants v23.12.2 (/home/mbp/src/mutants)
Finished release [optimized] target(s) in 4.49s
Running `target/release/cargo-mutants mutants --shard 1/10 --test-tool=nextest`
Found 56 mutants to test
ok Unmutated baseline in 13.9s build + 22.3s test
Auto-set test timeout to 1m 51s
56 mutants tested in 4m 22s: 40 caught, 16 unviable
; c r --release mutants --shard 1/10
Finished release [optimized] target(s) in 0.08s
Running `target/release/cargo-mutants mutants --shard 1/10`
Found 56 mutants to test
ok Unmutated baseline in 13.8s build + 21.7s test
Auto-set test timeout to 1m 48s
56 mutants tested in 3m 46s: 40 caught, 16 unviable
Although it's probably better at stopping early, it may be paying a higher per-test overhead or maybe making less use of parallelism?
Performance on stackslib
I also tested this on a 1% sample of stackslib with
; c r --release mutants --shard 1/100 -d ~/src/stacks-core/ --test-tool nextest
Finished release [optimized] target(s) in 0.18s
Running `target/release/cargo-mutants mutants --shard 1/100 -d /home/mbp/src/stacks-core/ --test-tool nextest`
Found 112 mutants to test
ok Unmutated baseline in 116.1s build + 244.5s test
Auto-set test timeout to 20m 22s
MISSED src/burnchains/bitcoin/spv.rs:1150:55: replace > with == in SpvClient::get_target in 34.1s build + 256.1s test
MISSED src/chainstate/burn/db/sortdb.rs:2893:25: replace == with != in SortitionDB::is_db_version_supported_in_epoch in 25.8s build + 283.8s test
3358.435171577s INFO timeout after 1222.4s, terminating child process...
TIMEOUT src/net/connection.rs:1334:9: replace NetworkConnection<P>::send_data -> Result<usize, net_error> with Ok(1) in 34.8s build + 1222.4s test
MISSED src/chainstate/coordinator/comm.rs:105:9: replace SignalBools::receive_signal -> u8 with 0 in 38.0s build + 278.8s test
MISSED src/main.rs:957:16: replace == with != in main in 24.3s build + 258.5s test
MISSED src/net/p2p.rs:4235:9: replace PeerNetwork::need_block_or_microblock_stream -> Result<bool, net_error> with Ok(true) in 28.7s build + 267.8s test
MISSED src/net/p2p.rs:1280:9: replace PeerNetwork::num_peers -> usize with 1 in 26.8s build + 311.8s test
test src/clarity_vm/database/mod.rs:174:9: replace <impl HeadersDB for ChainstateTx<'a>>::get_burnchain_tokens_spent_for_block -> Option<u128> with Some(1) ... 233.0s
MISSED src/net/http.rs:1730:9: replace HttpRequestType::get_proof_query -> bool with true in 33.5s build + 291.0s test
MISSED src/util_lib/db.rs:656:20: replace > with == in tx_busy_handler in 35.5s build + 290.3s test
test src/chainstate/stacks/db/unconfirmed.rs:443:9: replace UnconfirmedState::num_mined_txs -> usize with 0 s
└ PASS [ 28.317s] blockstack-core clarity_vm::tests::costs::epoch_21_test_alt
47/112 mutants tested, 8 MISSED, 1 timeout, 24 caught, 14 unviable, 1h 49m 40s elapsed, about 2h 32m remaining
It looks like some of those tests are still pretty slow, but it is making progress.