cargo-criterion
cargo-criterion copied to clipboard
Passing `--measurement-time` works but returns error 101
When I use --measurement-time criterion it works as expected, but has an exit status: 101 and echo $? returns 1. But without itecho $? returns 0. I've used time so with you can see that "measurement-time" increased. See below for a little more info.
$ time cargo criterion -- --measurement-time=10
Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs
time: [48.865 ns 48.942 ns 49.027 ns]
change: [-4.7610% -4.3953% -4.0512%] (p = 0.00 < 0.05)
Performance has improved.
sm_enum_msgs_any/sm_enum_msgs_any
time: [62.483 ns 62.589 ns 62.693 ns]
change: [-5.8203% -5.0130% -4.3315%] (p = 0.00 < 0.05)
Performance has improved.
sm_individual_msgs_any/sm_individual_msgs_any
time: [45.092 ns 45.393 ns 45.704 ns]
change: [-3.6137% -2.9714% -2.2768%] (p = 0.00 < 0.05)
Performance has improved.
error: Unrecognized option: 'measurement-time'
Error: Non-Criterion.rs benchmark target exper_message_trait exited with exit status: 101
real 0m53.730s
user 1m14.721s
sys 0m0.818s
$ echo $?
1
$ time cargo criterion
Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs
time: [50.109 ns 50.196 ns 50.283 ns]
change: [+2.1998% +2.4297% +2.6668%] (p = 0.00 < 0.05)
Performance has regressed.
sm_enum_msgs_any/sm_enum_msgs_any
time: [63.337 ns 63.442 ns 63.566 ns]
change: [+1.6191% +2.0968% +2.5335%] (p = 0.00 < 0.05)
Performance has regressed.
sm_individual_msgs_any/sm_individual_msgs_any
time: [48.363 ns 48.574 ns 48.759 ns]
change: [+4.3939% +5.0563% +5.7090%] (p = 0.00 < 0.05)
Performance has regressed.
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
real 0m39.131s
user 1m1.939s
sys 0m0.796s
$ echo $?
0
If you interrupt the "interactive output" you can see "estimated 10.000 s (195M iterations)":
$ time cargo criterion -- --measurement-time=10
Finished bench [optimized] target(s) in 0.03s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 10.000 s (195M iterations)^C
real 0m5.631s
user 0m0.177s
sys 0m0.051s
Where as without --measurement-time we see "estimated 5.0001 s (97M iterations)":
$ time cargo criterion
Finished bench [optimized] target(s) in 0.02s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 5.0001 s (97M iterations)^C
real 0m4.583s
user 0m0.152s
sys 0m0.058s