Use check to remove entries

[blairj09]

The check parameter currently functions as a mechanism to check for some equivalence across function results. If this criteria is not met, microbenchmark() returns an error. It would be nice if check could be extended to remove results which didn't meet a specific criteria rather than throw an error and stop the entire benchmarking process.

Ideally in this scenario, once a function fails a test, it is no longer run while the other functions continue running as expected.

Current Behavior

library(microbenchmark)
my_check <- function(values) {
  all(sapply(values[-1], function(x) identical(values[[1]], x)))
}

f <- function(a, b)
  2 + 2

a <- 2
## Check passes
microbenchmark(2 + 2, 2 + a, f(2, a), f(2, 2), check=my_check)
#> Unit: nanoseconds
#>     expr min    lq   mean median    uq  max neval
#>    2 + 2  73 102.5 129.81  124.0 140.5  307   100
#>    2 + a 103 145.0 210.22  160.0 175.5 4545   100
#>  f(2, a) 180 281.0 334.76  315.5 371.5  671   100
#>  f(2, 2) 183 285.0 356.71  310.5 341.0 4013   100
a <- 3
## Check fails
microbenchmark(2 + 2, 2 + a, f(2, a), f(2, 2), check=my_check)
#> Error: Input expressions are not equivalent.

Suggested Behavior

library(microbenchmark)
my_check <- function(values) {
  all(sapply(values[-1], function(x) identical(values[[1]], x)))
}

f <- function(a, b)
  2 + 2

a <- 2
## Check passes
microbenchmark(2 + 2, 2 + a, f(2, a), f(2, 2), check=my_check)
#> Unit: nanoseconds
#>     expr min    lq   mean median    uq  max neval
#>    2 + 2  73 102.5 129.81  124.0 140.5  307   100
#>    2 + a 103 145.0 210.22  160.0 175.5 4545   100
#>  f(2, a) 180 281.0 334.76  315.5 371.5  671   100
#>  f(2, 2) 183 285.0 356.71  310.5 341.0 4013   100
a <- 3
## Check fails
microbenchmark(2 + 2, 2 + a, f(2, a), f(2, 2), check=my_check)
#> Unit: nanoseconds
#>     expr min    lq   mean median    uq  max neval
#>    2 + 2  73 102.5 129.81  124.0 140.5  307   100
#>  f(2, 2) 183 285.0 356.71  310.5 341.0 4013   100

[joshuaulrich]

Thanks for the suggestion! I'm a bit reluctant to change the default behavior, since some users may rely on error-handling they've created. But I've thought about adding a checkmark() function that would check results by default. That new function could warn by default, but continue running. Thoughts?

[blairj09]

I like it! I agree about not changing default behavior.