Support for running tests made with the tinytest package

[atheriel]

Is your feature request related to a problem? Please describe.

Currently, devtools::test() is tied to the testthat package, which is sensible enough. However, there are more than a hundred CRAN packages that use the tinytest package instead (which will likely grow), and it's unfortunate that they cannot actually run them with devtools workflows. Moreover, the RStudio testing UI uses devtools::test(), so these widgets will also fail to detect tinytest test suites.

Describe the solution you'd like

I'd like to propose adding support for detecting and running tinytest test suites to devtools::test(). I'm fairly sure this can be done and could try to implement it myself if there is interest. It's possible that some of the other testing functions could support tinytest as well.

Describe alternatives you've considered

The tinytest package obviously provides its own tools to run tests, but I'm after the broader integrations here.

[hadley]

We discussed this in depth internally. We have two major reservations:

1. Generally, adding support for development methodologies that we don't personally use hasn't been particularly successful: we don't understand the usage deeply enough to design a strong integration with devtools, and because we don't use it, it inevitably breaks over time and it takes us a long time to fix it. This creates a bad user experience, and it's easy for the user to blame the other package.

2. As soon as devtools supports two testing frameworks, there will be considerable pressure to support more (e.g. RUnit, testit, ...), which causes the problem in 1. to compound.

However, I think there's a possible compromise — instead of adding support for tinytest, you could add support for all non testthat testing frameworks, but implementing something that mimics what R CMD check does, i.e. load the package and then source each file in tests. Overall, the key challenge will be balancing performance with correctness (i.e. closeness to R CMD check), and you can only determine where to sacrifice correctness empirically by trying it out with real code:

• How should the package be loaded? load_all() would be fastest, but installing the package into a temporary library would be closer to what R CMD check does. Will using load_all() cause problems in practice?
• How should the files be run? source()ing file in a temporary environment would be easy, but R CMD check actually runs each test in a separate process in the global env. Is that difference likely to cause problems in practice?

If you were interested in having a go at implementing this more general strategy, we'd be happy to discuss approaches and review a PR. Since we only use testthat for our tests, we'd be relying on you to try it out on a few packages, and see if it provides a smooth workflow.

[atheriel]

Thanks, that's a very thoughtful response. I think emulating R CMD check is likely to be a good approach as well. I'll see if I can get something working when I have the time, unless someone beats me to it.