[feature suggestion / idea] parameterized tests ?

[eddelbuettel]

Would it be useful to borrow some ideas from other languages and test framework to automagically loop over sets of parameters? Python's has this 'parametrize' for pytest which has been pointed out as useful. Is that something we can borrow?

[eddelbuettel]

Also: https://github.com/google/patrick which seems to do this for testthat

[michaelquinn32]

In principle, it shouldn't be a problem to create a fork of patrick to work with tinytest instead of testthat; let's call it tinypatrick. Generally speaking, patrick is a very simple package, generating test cases within a loop. It doesn't seem like tinytest use that concept, which is fine, but we would need to understand

• What happens when assertions run in a loop
• If failures should early exit
• If failures still print meaningful values
• What the most meaningful level of a loop might be; should we introduce something like a test case or should it happen at the file level

Feel free to explore if you've got the time. Otherwise, I might be able to put something together soon.

Best wishes, Michael

[eddelbuettel]

Thanks so much for piping in here!

tinytest works quite well via extensions, I once did one myself (ttdo) a while back. And I had forgotten too (but kicked the can a little) that @vincentarelbundock had put something in to allow mulitple extensions at once. And while I have to admit that I have not yet looked in detail into patrick, I noticed that it is short-ish in its main function which is encouraging :)

R being R, and allowing for so many tricks computing on the table of which @markvanderloo has deployed quite a few I have the feeling that is both something we all should do, and could. Then again, takes someone with an itch to scratch and a bit of time...

I'll let @markvanderloo answer the mechanics. There is something truly clever going on which is describes in a pair of papers at his arXiv page. Maybe we find a way to "shimmy" the implicit loop in there.

[markvanderloo]

I think that you can already do much of this with little effort because it is possible to program over tests.

For example, you can already do this in tinytest:

addOne <- function(n) n + 1

inputs <- 1:5
outputs <- 2:6
mapply(expect_equal, addOne(inputs), outputs)

Or, you can explicitly loop over tests, as for example in this test file from the wand package. Here, Bob creates a list of input-output pairs and loops over that.

But I might be misunderstanding the question here, please tell me if I do.

[eddelbuettel]

Yes, I also explicitly loop in some test scripts. I think the idea would be to show / document / help. The mapply example is compelling, question is about generalizing to non-scalar / non-simple results. However, as mapply can consume lists maybe this is moot ... or just a question of a new section in the vignette?

[markvanderloo]

Yes, it would be a good idea to document 'parameterized tests' in the vignette. Perhaps in the 'tinytest by example' vignette. (I also should move those to simplermarkdown)

[eddelbuettel]

One problem with the mapply example is that the feedback in case of error is a little convoluted. If I replace one of the output values with NA I get a fairly convoluted call stack:

$ tt.r -f param.R     ## tt.r is a simple littler wrapper, using run_test_file with -f file
param.R.......................    5 tests 1 fails 52ms
----- FAILED[data]: param.R<8--8>
 call| mapply(expect_equal, addOne(inputs), outputs)
 call| -->c("(function (...) ", "{", "    out <- fun(...)", "    if (inherits(out, \"tinytest\")) {", "        attr(out, \"file\") <- env$file", "        attr(out, \"fst\") <- env$fst", "        attr(out, \"lst\") <- env$lst", "        attr(out, \"call\") <- env$call", "        attr(out, \"trace\") <- sys.calls()", "        if (!is.na(out) && env$lst - env$fst >= 3) ", "            attr(out, \"call\") <- match.call(fun)", "        env$add(out)", "        attr(out, \"env\") <- env", "    }", "    out", "})(dots[[1]][[5]], dots[[2]][[5]])"
 call| )
 diff| Expected 'NA', got '5'
 
Showing 1 out of 5 results: 1 fails, 4 passes (52ms)
$ 

Maybe that's an auxiliary problem and one that's not easy to avoid

[eddelbuettel]

Next question: Michael has a nice and compact 'trigonometrics' example on the README.md of patrick (and in the docs). Foregoing the tribble, we can build a simple list of (a)lists:

ll <- list(alist("sin", sin(pi/4), 1/sqrt(2)),       
           alist("cos", cos(pi/4), 1/sqrt(2)),        
           alist("tan", tan(pi/4), 1)) 

A very pedestrian of way of working with this is looping and evaluating:

eval_expect_equal <- function(ll) {
    for (l in ll) {
        v <- eval(l[[2]])
        w <- eval(l[[3]])
        val <- expect_equal(v, w, info=l[[1]])
    }
}
eval_expect_equal(ll)

which gets us the usual aggregation:

$ tt.r -f /tmp/ttdo-prs/param.R 
param.R.......................    3 tests OK 41ms
All ok, 3 results (41ms)
$

Can you think of simpler / nicer / better way to unrool that we could 'hide' in a helper function (to which we may pass the predicate, expect_equal, too?

[michaelquinn32]

This is pretty similar to the interface in patrick. We should be able to do basically the same thing, without tidy* helpers.

with_parameters <- function(assertions, ..., .calling_env = parent.frame()) {
  params <- list(...)
  results <- vector("list", length(params))
  for (i in seq_along(params)) {
    results[[i]] <- eval(
      substitute(assertions),
      params[[i]],
      enclos = .calling_env
    )
  }
  results
}

x <- 3

with_parameters(
  {
    tinytest::expect_equal(expr, numeric_value, info = info)
  },
  sin = list(expr = sin(pi / 4) - x, numeric_value = 1 / sqrt(2) - 3, info = "sin"),
  cos = list(expr = cos(pi / 4), numeric_value = 1 / sqrt(2), info = "cos"),
  tan = list(expr = tan(pi / 4), numeric_value = 1, info = "tan")
)

#> [[1]]
#> ----- PASSED      : <-->
#>   call| eval(substitute(assertions), params[[i]], enclos = .calling_env)
#> info| sin 
#> 
#> [[2]]
#> ----- PASSED      : <-->
#>   call| eval(substitute(assertions), params[[i]], enclos = .calling_env)
#> info| cos 
#> 
#> [[3]]
#> ----- PASSED      : <-->
#>   call| eval(substitute(assertions), params[[i]], enclos = .calling_env)
#> info| tan  

This doesn't play especially nice with the current expression parsing in tinytest, but I think it's something that could be fixed.

[eddelbuettel]

I should putz around more "on the language". I suspected I could help myself with a reference to parent.env. This is gettting somewhere -- very nice second step.