How to write comprehensive unit tests?

[leeper]

One issue I've been wrestling with is how to write effective, comprehensive unit tests (regardless of particular testing framework). This remains a sticking point for convincing people to switch from "reliable" commercial products to "no warranty" open-source analytics. With the advent of code coverage as a metric, the push can be to create 100% coverage but coverage per se does not indicate correctly functionality. So what can we do to make credible open-source software?

We can think of it like this:

1. R CMD check ensures code can be parsed and is documented
2. Examples and unit tests ensure all (tested) code can be evaluated
   • Here is where code coverage might not imply much about actual functionality
3. Complete unit tests with expectation statements ensure code returns correct return value for one (possibly default) set of inputs
4. Comprehensive unit tests ensure all code returns correct return value for all combinations of inputs

For beginners, it is a lot of work to pass 1. For a complete package, it is also a lot of work to achieve 2 (and coverage stats suggest even really well-developed score pretty low on this measure). Can we develop guidance and/or tools that help developers get from step 2 to step 4?

[richfitz]

For tooling, something like Haskell's QuickCheck perhaps? I believe there are a few things that try to do this for R (but not sure if any are on CRAN yet). In this vein Python has hypothesis which is apparently quite nice.

[richfitz]

Ah, here's the R quickcheck package I was thinking of: https://github.com/RevolutionAnalytics/quickcheck

[maelle]

Guidance and also arguments and examples about why it's worth it? I've already heard "I write good code, why would I need unit tests?".

[jimhester]

I am not sure 4 is actually a worthwhile goal and quickly becomes computationally infeasible for all but the most trivial of functions. QuickCheck et. al. are useful to approximate 4. over time but I think more needs to be done to increase their usage in the R community.

Coverage Is Not Strongly Correlated With Test Suite Effectiveness is a interesting paper on the topic which suggests that coverage is uninformative after controlling for test suite size.

[craigcitro]

I'm also interested to know more about what people are up to here, and offer this blog post by Dan Luu as a worthwhile motivational read.

[nicolewhite]

Any interest in behave-style tests? I always miss this workflow when I'm working in R; testthat only covers unit tests, but often I have a collection of functions that work together and I want to write a feature test for them. I can hack it together in testthat more or less, but I wonder if it's worth the effort to develop an API specifically for feature tests.

[sckott]

often I have a collection of functions that work together and I want to write a feature test for them.

+1 - would be nice to easily do these integration tests to make sure functions/etc. work together as expected

[leeper]

Here's language for a pull request:

As a result of a discussion at the rOpenSci unconf, we propose the addition of a @tests tag to roxygen2 that allows in-line tests that are essentially roxygenized to a /tests directory. An example of this would be:

#' @tests
#' context("Example tests")
#' test_that("A test", {
#'   expect_true(TRUE)
#' })

This PR modifies roxygen2::roxygenize() to automatically generate a .R file in /tests/testthat and, if necessary, setup the /tests/testthat.R infrastructure. (We intend to only initially propose support for testthat, but this could be generalized to support other testing frameworks, such as RUnit, Bioconductor, or testit, etc.)

The reason for the addition of this roxygen feature follows the same spirit as the in-line documentation features already provided by royxgen: it makes sense to write tests near the definition of a function. In some sense, examples are tests (e.g., uses of stopifnot()) but this provides a way to produce more comprehensive tests in an roxygen-like way.

We suspect that this will feature encourage package developers to write tests by dramatically reducing the effort needed to produce a test suite. It may make sense to add additional features such as a @test tag that (ala @example) allows the import of tests from another file, or some kind of summary documentation of what functions have tests.

[jimhester]

just found (https://github.com/klutometis/roxygen/issues/88) which is actually pretty close to what @leeper is proposing. Might not be too hard to update to current roxygen.