Ranking seems reversed

[aadler]

trafficstars

Description

In the example I am testing (stimulated by this SO question, the function which takes less time is ranked 'b' instead of 'a'.

Expected behavior

That the fastest performing snippets get letters closer to the start of the alphabet.

Minimal, reproducible example

library(microbenchmark)
n <- 6
alpha <- 1:5
out <- function(x, n) outer(x, seq_len(n) - 1, `^`)
mat <- function(x, n) matrix(rep(x, each = n) ^ (seq_len(n) - 1),
                             ncol = n, byrow = TRUE)
microbenchmark(out(alpha, n), mat(alpha, n), check = 'identical',
               control = list(order = 'block'), times = 10000L)

Unit: microseconds
          expr min  lq    mean median  uq    max neval cld
 out(alpha, n) 5.4 6.1 8.61506    7.2 8.7 2264.4 10000  a 
 mat(alpha, n) 3.0 3.2 3.91958    3.3 3.5 2121.1 10000   b

Session Info

R Under development (unstable) (2023-06-22 r84597 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 10 x64 (build 19045)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: Asia/Jerusalem
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] microbenchmark_1.4.10

loaded via a namespace (and not attached):
 [1] MASS_7.3-60      TH.data_1.1-2    zoo_1.8-12       compiler_4.4.0   Matrix_1.5-4.1   multcomp_1.4-25 
 [7] sandwich_3.0-2   tools_4.4.0      survival_3.5-5   mvtnorm_1.2-2    codetools_0.2-19 splines_4.4.0   
[13] grid_4.4.0       lattice_0.21-8  

[joshuaulrich]

Thanks for noticing this! It's intended behavior. The default printed sort order is the order the expressions are passed to microbenchmark(). And "b" will be first if you order by "cld" because the two values are actually "a " and " b" and the space sorts first.

You can change the printed sort order using the order argument to print.microbenchmark().

library(microbenchmark)
n <- 6
alpha <- 1:5
out <- function(x, n) outer(x, seq_len(n) - 1, `^`)
mat <- function(x, n) matrix(rep(x, each = n) ^ (seq_len(n) - 1),
                             ncol = n, byrow = TRUE)
mb <- microbenchmark(out(alpha, n), mat(alpha, n), check = 'identical',
               control = list(order = 'block'), times = 10000L)
print(mb, order = "median")

The "cld" column doesn't represent the order of expression speed. I don't know exactly what it represents, but it's calculated by multcomp::cld().

This could be documented better in ?microbenchmark and ?print.microbenchmark. I'd appreciate it if you would suggest some changes that would clarify the behavior. Especially if you have any clue what "cld" means.

[aadler]

Thank you, Joshua. I've always understood the letter-mapping to reflect "best" to "worst" for code snippets whose timings are statistically significantly different. In every one of my thousands of calls to microbenchmark this has been the case, until now. The help for the print method implies this understanding as well given the sentence "If the multcomp package is available a statistical ranking is calculated and displayed in compact letter display from in the cld column."

Digging a little deeper, it's calling multcomp::glht(mdl, multcomp::mcp(expr = "Tukey")) inside summary since the default is true. mdl is lm(time ~ expr, object) which strikes me as a little weird give each expression is unique, is it not?

This is not my area of expertise, but it seems to me that what @olafmersmann intended was to use the multcomp package to pairwise test the all the expressions' means against each other using glht and "binning" them into "statistically indistinguishable" categories. He used cld to extract the bins. I haven't dug far enough into the code of cld itself to see if the letter assignations are actually rankings or merely mappings. If the latter, that would explain the above result. If we can figure this out, it may pay to update the documentation.

Also, the way I use microbenchmark often is to just call it and rely on the auto-printing to the screen. This means I can neither control the print order nor turn off include_cld since the former only lives in the print method and the latter only in the summary method. To engage that behavior I would have to call print(summary(mb, include_cld = FALSE), order = 'median').

However, for my example above, that still does not seem to work:

library(microbenchmark)
n <- 6
alpha <- 1:5
out <- function(x, n) outer(x, seq_len(n) - 1, `^`)
mat <- function(x, n) matrix(rep(x, each = n) ^ (seq_len(n) - 1),
                             ncol = n, byrow = TRUE)
mb <- microbenchmark(out(alpha, n), mat(alpha, n), check = 'identical',
                     control = list(order = 'block'), times = 10000L)

print(summary(mb, include_cld = FALSE), order = 'median')

           expr min  lq    mean median  uq    max neval
1 out(alpha, n) 5.3 5.8 7.07789    6.8 7.2 2077.0 10000
2 mat(alpha, n) 3.0 3.2 4.60786    3.3 3.7 5461.2 10000