I have been working on Android for a while now and have spent a lot of that time working on various parts of the Android testing infrastructure. As part of that work I noticed a few issues with the support that JUnit has for integrating into different environments. In order to address those limitations I would like to make a number of changes:

• Make JUnit 3 tests run more like JUnit 4 tests, use Statement and ParentRunner where possible and fallback to JUnit38ClassRunner only when required to maintain backwards compatibility.
• Add a rule that is generic like TestRule but is passed the object like FrameworkMethod to make it as widely and generally useful as possible; call it TargetedTestRule
• Add support for global parameters, call them RunnerParams that can be used to customize and configure JUnit behavior.
• Add support for globally configured TargetedTestRule to be applied to all tests being run.

One thing to bear in mind is that the large majority of Android tests are JUnit 3 tests, i.e. extend TestCase. I'm talking around ten thousand classes, with hundreds of thousands of individuals tests. Many tests are imported from upstream sources and so modification of the tests themselves is not a practical solution.

Android Testing Overview

Android testing has a number of features not provided by JUnit:

1. A global default timeout that is used for any test that does not have its own @Test(timeout) set.
2. A dry run mode where events are fired as if the tests were run and passed but without actually running the tests.
3. Injection of environment specific objects into tests.
4. Lazy creation of TestCase objects so that they are created just before the test is started.
5. Early cleanup of the TestCase objects so that they are released just after the test has finished.
6. Defer validation of methods until as late as possible. This ensures that as many methods as possible are run. JUnit will not run any methods in a JUnit 4 test class if even one is invalid.
7. The global timeout is a pragmatic solution. A lot of the tests that are run do not belong to Android but rather they are pulled from an upstream source and there is just not enough time to go through them all and specify a timeout. The tests are also run within what are essentially untrusted environments as it is the passing of the tests that make the environment trustworthy. Having a global timeout can ensure that test runs complete in a reasonable amount of time which is important when trying to detect and debug lots of issues.

The dry run mode is used to allow the test runner to create a list of all the tests that will be run in advance so that it can track the run's progress, restart if necessary, etc. It's possible that it could get the same result by traversing the Description but custom implementations of junit.framework.Test could cause problems.

The annotation driven injection into JUnit 4 classes was removed because they could not get it to work consistently across all tests. However, it is still done for JUnit 3 tests. Instead they use a global registry which is nowhere near as clean as using method injection.

The lazy creation and early release are important to minimise the size of the memory footprint over the lifetime of the test run. In some cases not having it will result in OOM and fail the whole run. It's not an issue for JUnit 4 based tests as the lifespan of test objects is already limited to the duration of a single test.

The AndroidJUnitRunner (AJUR) class is the main entry point for all this code and it uses the AndroidRunnerBuilder (extends AllDefaultPossibilitiesBuilder) indirectly to customize the JUnit behavior. The code has to jump through an awful lot of hoops in order to implement the above capabilities.

Vogar

There is another tool Vogar, which is a little lower level and used by developers of the core Android libraries. At the moment it has its own JUnit implementation (that implements a subset of JUnit 3/4 in a slightly different way). The main reason that custom implementation was created was to do lazy creation/early release of the TestCase objects when running JUnit 3.8. I am working on moving it over to using standard JUnit 4.10.

Other users

There are more users that want global rules or at least capabilities that can easily be implemented using global rules. e.g.

• Gradle asked for a global timeout (https://discuss.gradle.org/t/timeouts-in-junit-tests/318).
• Global JUnit rules revisited (https://github.com/junit-team/junit4/issues/1219) - has details of other requests for global rules.

Based off those discussions I know that the JUnit team are loathe to add support for things like global timeout, global rules and while I can understand your reasoning I'm hoping that I can convince you that pragmatically they are necessary.

Global Rules

The first three features of the Android testing infrastructure can be easily supported once global rules are available.

1. The Timeout rule already provides a test class specific timeout.
2. A dry run can simply be a rule that returns a Statement that does nothing.
3. Injection can be done by a TargetedTestRule as it would have access to the test object.

The remaining changes will require specific changes within JUnit and the last one cannot be done by default, it would have to be a configurable option.

These changes would allow AJUR (and other users) to customise JUnit behavior a lot more easily, more consistently and more generally that it can at the moment. e.g. It cannot handle any custom Runners that it does not yet have explicit support for because there is no common way to request a timeout or dry run. A built in global rule mechanism would make it possible for providers of custom Runner classes to integrate properly into the Android testing environment.

None of these changes is intended to change the behavior of tests, other than as required to ensure that the tests run correctly. Similarly, they should not prevent tests being run inside an IDE unless they cannot be run inside an IDE. e.g. A test that requires say a real instance of android.content.Context to be supplied cannot run directly within an IDE, it must run within an Android environment, which could itself run within an IDE. If global rules/parameters are supported then IDEs will add support for configuring their test runners in the appropriate way.

Global Parameters

Global parameters will make it possible to change JUnit behavior in ways that are backwardly incompatible by making the new behavior optional which gives the junit team a lot more flexibility in what and how they do it.

JUnit 5

I understand that JUnit 4 is largely in maintenance mode and has no new features planned and that the focus in on JUnit 5. However, there is no way that we can move to JUnit 5 anytime soon (I'm not sure if it can actually run JUnit 3 tests) and we have a huge amount of testing infrastructure code that depends on JUnit 4. These changes are just part of a (for now) speculative plan to iteratively improve Android's testing infrastructure.

1. Switch Vogar over to depending on JUnit rather than have it roll its own, that will allow us to use JUnit 4 tests, including many tests from upstream that are not currently supported.
2. Upgrade version of JUnit that Android depends on from 4.10 to 4.12.
3. Migrate tests off JUnit 3 to JUnit 4 - specifically to allow use of TestRules in order to reuse the logic and make the tests more robust.
4. Allow tests to use custom Runners, such as JUnit Params; at the moment they are limited to the ones explicitly supported by AJUR itself, using other ones will result in them at the very least being run twice.

If it helps I'd be quite happy to do the grunt work of maintaining JUnit 4.x along with these changes.

Current Changes

I have done most of the prototyping work already, see https://github.com/paulduffin/junit/tree/wip-junit-enhancements for details. Obviously, nothing that I've done is set in stone.

They can roughly be grouped into the following:

• Add Parameterized high level tests to allow the 'runner' to be supplied as a parameter allowing easy verification of new ways to run the tests and changes in the runner structure.
• Avoid using JUnit38ClassRunner where possible. Instead convert TestCase/TestSuite instances directly into Runner instances that use Statement. JU38CR is only used for those tests whose behavior depends on a TestResult.
• Add a global RunnerParams object for passing a type safe, immutable collection of options into the runners. This opens up the possibility for changing the behavior of JUnit without breaking existing users.
• Leverage the RunnerParams to allow some aspects of JUnit's behavior to be customised. e.g. defer method validation until methods are invoked.
• Add a new TargetedTestRule, abstract and generally applicable like TestRule but passed a reference to the test object like FrameworkMethod. Add TargetedRuleChain which can take both TargetedTestRule and TestRule.
• Add a global option to specify a TargetedTestRule and apply that to all Statements, including the ones created for JUnit 3 based tests. JU38CR will fail if an attempt is made to use it in combination with a non-identity TargetedTestRule because the rules cannot be applied to custom Tests and so it would not be safe to just run the tests.

Beware that I often rebase/force push to clean up the history and fix issues that I find while working on later changes.

@paulduffin, when you say "there is no way that we can move to JUnit 5 anytime soon (I'm not sure if it can actually run JUnit 3 tests)," what is it that keeps you from moving to JUnit 5 for the runtime?

The JUnit 5 framework (i.e., the Launcher and TestEngine APIs) have to be launched on a Java 8 JVM, but that does not pose a hurdle with regard to what types of tests can be executed. A TestEngine can in fact execute tests compiled on previous versions of the JVM (e.g., Java 5, 6, 7). Furthermore, the JUnit4TestEngine perfectly runs tests based on either the JUnit 4 or JUnit 3 programming models.

Thus, to answer your parenthetical question: yes, JUnit 5 can run JUnit 3.8 tests without a problem.

Regarding your proposal for introducing RunnerParams, the JUnit 5 Launcher and TestEngine APIs already support ConfigurationParameters which achieve the same goal.

There have also been requests to support global timeouts in JUnit 5, and I believe there is a good chance that such a feature will make it into JUnit 5 before the GA release.

@sbrannen Basically, we have an awful lot of code involved in running the JUnit tests (i.e. not the tests themselves) which is dependent on JUnit 4 classes. Moving straight from that to using JUnit 5 is not really viable, especially if it doesn't yet have the capabilities that we need.

Global timeouts and parameters aren't sufficient, ideally we need the ability to define the equivalent of global TestRules. Are Extensions the equivalent in JUnit 5?

Android has some support for Java 8 but it's not complete so I'm not sure if we'll even be able to run JUnit 5 at all until we improve support for Java 8.

Having said that supporting JUnit 5 would obviously be a longer term goal.

Does anyone have any objections to extending JUnit 4.13/14 as requested here? I'd like to know whether I should continue working on this.

We have has many discussions about global.Rules and David in particular was concerned about test behavior being specified ouside of the test class. I am not saying we wouldn't support it, but it has been rejected in the past.

I personally think a global timeout would be fine if 1) the timeout exception made it clear that the timeout was global , 2) the timeout could be overridden in the test, 3) you get the timeout if the test was run in the IDE (which would be thei case If the global timeout was specified in a package-level annotation). A timeout set in a system property would be another option, if the timeout exception gave a clue as to how to enable the timeout in an IDE.

I don't think global rules would give you a reliable dry run mode.. For starters , not all runners support Rules. Also, your dry run would execute class rules. Can you explain the cases when looking at the Description tree doesn't work?

WRT your restrictions on global timeouts.

1. Ok, given that this would just be a rule the rule could do anything. It would largely be up to the person configuring the timeout rule.
2. As long as the test can only specify a shorter timeout that would be fine. Otherwise, if the test can specify a longer timeout or even no timeout that would pretty much eliminate a lot of the point of having a global timeout which is to protect the code running the tests (e.g. Android CTS tests) from badly written tests or badly behaving tests.
3. Requiring the timeout to be specified in a package level annotation would again eliminate most of the point of having it.

In Android there are three different actors (are often from different companies), none of which can 'trust' the other:

1. The Compatibility Test Suite (CTS) authors (the Android team and other upstream test writers).
2. The CTS test runner team.
3. The device manufacturer who is attempting to certify their device as Android compatible by passing the Android CTS.

Note, by lack of trust I don't mean that anyone is doing anything malicious just that it may not work. Until CTS has successfully passed the device (software/hardware) is not trusted. The test authors cannot predict all the issues that may arise on a new platform, e.g. a test that has previously always run to completion suddenly fails to complete due to a bug on the device. Similarly, the CTS infrastructure may not work properly.

Modifying the tests to add a global timeout isn't practical as it may need to change from one run of CTS to another, e.g. if the global timeout aborts a test then it may be run again with a bigger time limit to see if it is hung or simply taking longer than expected.

The purpose of these changes is to make it easier/possible to integrate it into a separate environment, e.g. Android. In that sense an IDE is just another environment. Tests that require to be run in Android cannot be run from within an IDE unless the IDE has specific support for running tests in Android. In that case it would need to use the Android runner that provides the global timeouts.

There's nothing to stop IDEs adding in support to allow a developer to specify global rules/timeouts to apply across all tests. Just a quick look at IntelliJ IDEA shows that it has nearly a dozen different options to control how tests are run, changing any one of them could change the way tests are run.

The class rules are a problem with dry run. At the moment they are still run in dry run mode but that is not a big issue because as I mentioned most of the tests are JUnit 3.8 based so don't support class rules. I don't know of any specific issues with just using the Description hierarchy instead of a dry run mode. It may be a hold over from when it used JUnit 3.8. If there were going to be any issues I suspect that it would be with parameterized tests. I will take a look though and see whether we can use the Description hierarchy instead.

Global test rules would allow us to replicate our existing dry run behavior very easily and have other uses such as timeouts, guards (as described in another issue), method/field injectors.

As for Runners not using Rules you are correct and that is part of the reason why a large part of the changes I made in my prototype involved changing all Runners so that they ran individual tests as Statement objects so that they could use Rules. The only one that needed work was JUnit38ClassRunner. While it's not possible to avoid using that for custom implementations of Test/TestCase/TestSuite for most common usages of TestCase/TestSuite it is possible to switch to using separate Runner/ParentRunner for them that use Statement and which preserve backwards compatibility. That also has a number of other benefits, e.g. filters and sorters are applied consistently, TestCase objects only exist for the duration of the individual test rather than the whole test run. For those cases where JUnit38ClassRunner is required for backwards compatibility any attempt to use global rules cause those tests to fail.