cargo-criterion icon indicating copy to clipboard operation
cargo-criterion copied to clipboard

Passing `--measurement-time` works but returns error 101

Open winksaville opened this issue 2 years ago • 0 comments

When I use --measurement-time criterion it works as expected, but has an exit status: 101 and echo $? returns 1. But without itecho $? returns 0. I've used time so with you can see that "measurement-time" increased. See below for a little more info.

$ time cargo criterion -- --measurement-time=10
    Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs                                                                             
                        time:   [48.865 ns 48.942 ns 49.027 ns]
                        change: [-4.7610% -4.3953% -4.0512%] (p = 0.00 < 0.05)
                        Performance has improved.

sm_enum_msgs_any/sm_enum_msgs_any                                                                             
                        time:   [62.483 ns 62.589 ns 62.693 ns]
                        change: [-5.8203% -5.0130% -4.3315%] (p = 0.00 < 0.05)
                        Performance has improved.

sm_individual_msgs_any/sm_individual_msgs_any                                                                             
                        time:   [45.092 ns 45.393 ns 45.704 ns]
                        change: [-3.6137% -2.9714% -2.2768%] (p = 0.00 < 0.05)
                        Performance has improved.

error: Unrecognized option: 'measurement-time'
Error: Non-Criterion.rs benchmark target exper_message_trait exited with exit status: 101

real    0m53.730s
user    1m14.721s
sys     0m0.818s

$ echo $?
1

$ time cargo criterion
    Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs                                                                             
                        time:   [50.109 ns 50.196 ns 50.283 ns]
                        change: [+2.1998% +2.4297% +2.6668%] (p = 0.00 < 0.05)
                        Performance has regressed.

sm_enum_msgs_any/sm_enum_msgs_any                                                                            
                        time:   [63.337 ns 63.442 ns 63.566 ns]
                        change: [+1.6191% +2.0968% +2.5335%] (p = 0.00 < 0.05)
                        Performance has regressed.

sm_individual_msgs_any/sm_individual_msgs_any                                                                             
                        time:   [48.363 ns 48.574 ns 48.759 ns]
                        change: [+4.3939% +5.0563% +5.7090%] (p = 0.00 < 0.05)
                        Performance has regressed.


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


real    0m39.131s
user    1m1.939s
sys     0m0.796s

$ echo $?
0

If you interrupt the "interactive output" you can see "estimated 10.000 s (195M iterations)":

$ time cargo criterion -- --measurement-time=10
    Finished bench [optimized] target(s) in 0.03s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 10.000 s (195M iterations)^C

real    0m5.631s
user    0m0.177s
sys     0m0.051s

Where as without --measurement-time we see "estimated 5.0001 s (97M iterations)":

$ time cargo criterion
    Finished bench [optimized] target(s) in 0.02s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 5.0001 s (97M iterations)^C

real    0m4.583s
user    0m0.152s
sys     0m0.058s

winksaville avatar Jan 12 '23 18:01 winksaville