Add SPI support (Service Provider Interface, from Java 6) to enable external RunListener

[rliesenfeld]

The @RunWith annotation has been widely successful in enabling the integration of other testing tools with JUnit. For example, the following testing libraries (among others) use it: jMock, Mockito, JMockit, PowerMock, Unitils, Spring Test.

However, this mechanism requires the annotation be applied to each and every test class, and does not support the use of more than one custom runner for the same test class.

I would like to propose a simple enhancement to JUnit's ability to integrate with other testing tools: add support for the automatic discovery of org.junit.runner.notification.RunListener implementations, through the standard Service Provider API available in Java SE 6 (the java.util.ServiceLoader class).

I am the creator of the JMockit testing toolking (http://code.google.com/p/jmockit), which already implements sophisticated integration with the JUnit test runner. However, to provide the best user experience I need JMockit to get initialized before JUnit starts running the first test. I have achieved this, but not in a clean way. The mechanism I propose would provide such a clean solution, which I believe might also be useful to other testing libraries which integrate with JUnit.

The desired SPI support can be achieved with the addition of the following four lines of code to the constructor of org.junit.runner.JUnitCore (alternatively, it could be done inside the RunNotifier class):

if ("1.6 1.7".contains(System.getProperty("java.specification.version"))) {
    ServiceLoader<RunListener> serviceLoader = ServiceLoader.load(RunListener.class);
    for (RunListener listener : serviceLoader)
        fNotifier.addListener(listener);
}

The code above requires Java 6, but it would not interfere with test suites compiled for and running under Java 5 (or older).

Thanks

[KentBeck]

Can you write a test for this?

[rliesenfeld]

Sure! (Although it has to be an integration test, given its nature.)

A jar file would have to be created, containing the two following files/dirs:

META-INF/services/org.junit.runner.notification.RunListener myTool/integration/junit/MyRunListener.class

The first is a text file with a single line, containing the name of RunListener implementation class: "myTool.integration.junit.MyRunListener" (without the quotes). The second is an arbitrary RunListener subclass.

Once you have this jar, the following JUnit test could be executed, with that jar in the classpath:

@Test
public void verifyMyRunListener() {
    assertTrue(MyRunListener.initialized);
}

The MyRunListener class could be implemented as (assuming the test class is in the same package):

public class MyRunListener extends RunListener {
    static boolean initialized;
    public MyRunListener() { initialized = true; }
}

In the particular case of JMockit, the jar containing the service would be jmockit.jar, and the RunListener implementation would be "mockit.integration.junit4.Initializer". I just implemented this in my environment and it works well (I edited a copy of the RunNotifier class and put it in the classpath before junit-4.8.1.jar). I ran my tests under the Sun JDK 1.6.0_18 and under JDK 1.5.0_19. Also, I didn't notice any reduction in the total test run execution time.

[KentBeck]

We wouldn't add this to JUnit without an automated test. I assume you can do all of the above steps directly in Java. If you do the translation into Java it will happen a lot sooner than if you wait for us to do it.

[rliesenfeld]

Ok, I will do what I can. Thanks.

[rliesenfeld]

Here is the automated test (100% Java with no external dependencies, it turned out 8^): package org.junit.tests.listening;

import java.io.*;
import java.net.*;
import org.junit.*;
import org.junit.runner.*;
import org.junit.runner.notification.*;
import static org.junit.Assert.*;

public final class ExternalListenerTest {
   public static final class CustomListener extends RunListener {
      boolean testRunStarted;
      public CustomListener() { externalListener = this; }
      @Override public void testRunStarted(Description description) { testRunStarted = true; }
   }

   private static CustomListener externalListener;

   public static final class OneTest {
      @Test public void nothing() {}
   }

   @BeforeClass
   public static void createServiceProviderConfigurationFile() throws Exception {
      URL location = CustomListener.class.getProtectionDomain().getCodeSource().getLocation();
      File file = new File(location.getPath() + "META-INF/services/" + RunListener.class.getName());
      file.getParentFile().mkdirs();

      Writer output = new FileWriter(file);
      output.write(CustomListener.class.getName());
      output.close();

      file.deleteOnExit();
   }

   @Test
   public void publishTestRunEventsToExternalRunListener() {
      JUnitCore core = new JUnitCore();
      Result result = core.run(OneTest.class);

      assertEquals(1, result.getRunCount());
      assertEquals(0, result.getFailureCount());
      assertEquals(0, result.getIgnoreCount());

      assertNotNull("External RunListener not loaded", externalListener);
      assertTrue("Missing event on external RunListener", externalListener.testRunStarted);
   }
}

Let me know if anything more is needed.

[rliesenfeld]

On further thinking, I am not sure if all development environments (such as Eclipse with the JUnit plug-in) use the JUnitCore class. Also, it's simpler and probably more appropriate to implement this enhancement in the RunNotifier class. So, the test above could instead be written as: @Test public void publishTestRunEventsToExternalRunListener() { new RunNotifier().fireTestRunStarted(null);

    assertNotNull("External RunListener not loaded", externalListener);
    assertTrue("Missing event on external RunListener", externalListener.testRunStarted);
}

[pniederw]

I'm the creator of Spock (http://spockframework.org), and have faced similar issues when integrating with JUnit (which is currently the only means for running a Spock test). It would be awesome if JUnit provided some "global" hooks like the one proposed here. Even better if such hooks worked even if JUnitCore isn't used. I've come across several environments which don't use JUnitCore, one example being the Gradle build system (http://gradle.org).

[pniederw]

Implementation note: I'd prefer to guard against the unavailability of ServiceLoader as follows:

try {
    ServiceLoader<RunListener> serviceLoader = ServiceLoader.load(RunListener.class);
    for (RunListener listener : serviceLoader)
        fNotifier.addListener(listener);
} catch (LinkageError ignored) {}

[rliesenfeld]

Hello again.

Any chance this enhancement will make it into JUnit someday? Notice I did provide an automated test (see previous comments), as requested.

TestNG already provides SPI support, since version 6.2. With that, the JMockit tool was able to provide fully transparent integration for TestNG users. It would be great to also provide this benefit to the JUnit users, which are the majority. More than that, with the additional enhancement of adding an "Object getTestObject()" method to org.junit.runner.Description, several other tools (like Mockito, Unitils, and Spring Test) would also be able to provide transparent integration with JUnit, saving users from having to add "@RunWith" or "@Rule" annotations in every test class.

[marcphilipp]

Would it be possible for any of you to work on a pull request for this feature?

[rliesenfeld]

I never used Git, but I will see what I can do. Thanks.

[rliesenfeld]

I just submitted a pull request: https://github.com/KentBeck/junit/pull/430

Let me know if anything else is needed. Thanks

[dsaff]

Can you describe in a little more detail what needs to happen in the initialization step? As mentioned in #430, the core of JUnit has not yet enabled any features that enable changes in behavior based on global state, so I'd like to convince myself that it would be necessary to change that policy in order to get what you're looking for.

[rliesenfeld]

I wrote this as a comment to the pull request, but it's probably better to repeat it here:

This is a mechanism to support certain kinds of JUnit extensions, without requiring users to do anything more than adding a jar file to the runtime classpath. Other special runtime environments, such as OSGi containers, Java EE, etc. already provide similar mechanisms, I believe. TestNG provides this exact same mechanism since version 6.2.

To me, it makes a lot of sense that a fundamental framework such as JUnit would support such a mechanism. The JUnit test runner is the entry point for the execution of a test run (well, apart from the software that calls into "JUnitCore", like Ant, Maven, IDEs, etc.), and there are many other testing tools which need to integrate somehow with the test runner. Today, users have to jump through unnecessary hoops (like adding "@RunWith" or a rule, or extending a base class) in order to make use of these extra tools in their test classes. The idea here is to provide a clean and non-intrusive solution.

[rliesenfeld]

For reference, the following article is relevant : http://java.sun.com/developer/technicalArticles/javase/extensible

Essentially, I would need JUnit to be more extensible, in order to enable other testing tools (such as my own and several others) to plug into the test runner in a way that is not intrusive to end-user test code. For example, test writers should be able to make use of a mock object injection library without having to add "noisy" extra code in their test classes - just annotating a field with (lets say) "@Mock" and another with "@Tested" should be enough. Does this sound reasonable?

[dsaff]

I'm understanding some of the motivation here, but in order to explore solutions, can you go into details about your specific situation? What needs to happen during the setup? Is there static state that needs to be initialized? Classes that need to be loaded? Understanding that will help us talk through the alternatives. Thanks.

[rliesenfeld]

In the case of my tool (JMockit), all it needs is to be called (with no input) at the beginning of a test run, before any tests are started. This is sufficient to activate the JMockit-JUnit integration, which depends on being notified of typical test execution events (test started, test finished, etc.) during the test run, in order to provide certain services such as the automatic instantiation of tested classes, the injection of mock objects, the verification of expectations, and general clean-up after a test or test class is completed.

Other tools (and also JMockit in the future, if/when the JUnit listener API gets richer) could do something similar. In the case of jMock 2 (another mocking tool), for example, the current "JMock" custom runner (see http://svn.jmock.codehaus.org/browse/jmock/trunk/jmock2/src/org/jmock/integration/junit4/JMock.java?r=1378) could be replaced with a RunListener implementation which simply called "Mockery#assertIsSatisfied()" after each test execution.

[dsaff]

I can guess this wouldn't be your preferred solution, but for my information, could you achieve your goals with a @Rule in each test class?

[rliesenfeld]

Yes, it would work, just like using @RunWith, extending a base test class, or calling some special method in a @BeforeClass method would. The problem with all these four alternatives (and they all are used today by several other tools which need to integrate with JUnit), is that they force end users to pollute their test classes with code that doesn't contribute anything to what the tests are meant to do. I mean, just because I want to have a "@Mock" field in my test class, I shouldn't have to add an strange "@Rule" field. The SPI-based solution unifies all these different solutions in a transparent, noise-free way.

[dsaff]

The trade-off is that, while it is "noise free", it is also "signal free". I can no longer look at a test case and know everything that's going on--I also have to know what all the jar manifests are.

With JUnit, we try to follow the rule that "all's fair if you pre-declare". That is, an extension can do whatever it wants, as long as there's some way to know what it's doing. I'm happy trying to get that declaration down to one line, and as short a line as we can make it, at that.

How high does removing that one last line of declaration code rank for your users?

[rliesenfeld]

I am not sure, but they are already used to not having to add it. I removed support for "@RunWith(JMockit.class)" some time ago, solving the problem in another way but with one small inconvenience: having to position jmockit.jar before junit.jar in the runtime classpath (an inconvenience the SPI support in JUnit would eliminate).

[dsaff]

I don't often hear feedback from users that want zero lines of configuration vs. one line. At this point, I'm really not inclined to introduce a feature that will encourage extensions and tests that will give right or wrong answers based on the Java version used in order to take out that single line of configuration.

I'm open to additional arguments, or to hearing that there's lots of user demand that hasn't gotten through to me, but that's where I stand at the moment.

[pniederw]

Note that the original proposal is about "JUnit's ability to integrate with other testing tools". In other words, this isn't a feature for people writing JUnit tests, but for people developing JUnit (runner) based testing tools. For many of these tools, the ability to observe the whole test run in an environment independent way (i.e. irrespective of who drives JUnit) is a crucial missing feature.

For my own JUnit-based testing framework (http://spockframework.org), the need to observe the whole test run comes up regularly. One example is generating a test report. For this one absolutely needs a "post test run" hook. Right now, Spock is simulating this with a JVM shutdown listener, which is an ugly and unreliably workaround. Other tools are forced to do the same (I believe http://www.unitils.org is one of them).

Please reconsider this feature request in the light of JUnit's increasingly important role as an interface between build tools/IDEs and JUnit (runner) based testing tools.

[dsaff]

I can understand wanting to add some kind of listeners at runtime. Can you help me understand a little more about how spock works on top of JUnit? Right now, RunListeners are added by JUnitCore. Is spock calling JUnitCore directly?

[pniederw]

Spock can't use JUnitCore since it isn't in a position to trigger test execution. Instead, Spock implements a @RunWith runner, hidden away in the spock.lang.Specification class that every test class needs to inherit from. This makes Spock compatible with all IDEs, build tools, and full-stack frameworks that support execution of JUnit tests. This is a huge benefit. Apart from that, Spock is its own testing framework, although with excellent JUnit compatibility (understands and leverages JUnit exceptions, JUnit rules work unchanged, Hamcrest support, etc.).

JUnit's Description class provides a JUnit view of a Spock test and can't capture all available information. Hence the desire to generate an enhanced Spock report in an environment-independent way, leveraging the same @RunWith integration point. Another example where an external RunListener is desirable is for a test rule that spans a whole test run, which is something that Spock already supports (albeit using workarounds like static variables and a JVM shutdown listener).

[Tibor17]

Due to -target 1.5 in junit, we can add a dependency netbeans openide util or use javax.imageio.spi.ServiceRegistry from 1.4.2.

I tested openide library. My service file /META-INF/services/service.IService has two classes in it Service Service2

here i can load both implementations: Lookup lookup = org.openide.util.lookup.Lookups.metaInfServices(ClassLoader.getSystemClassLoader()); Collection<? extends IService> c = lookup.lookupAll(IService.class);

In Maven this library means artifact: org.netbeans.api : org-openide-util-lookup : RELEASE70-BETA : jar http://bits.netbeans.org/maven2/org/netbeans/api/org-openide-util-lookup/RELEASE70-BETA/

[Tibor17]

@pniederw One year ago i started coding this solution with new annotation declaring the listener class on test class, or SPI without default listener. But this annotation would be junit based, and this is what you do not want. Making junit framework so dynamic would require you to design all the time a new runner and implementing RunNotifier because we do not have SPI for run listener. Then the RunNotifier would have to delegate to the SPI listeners. Even if we had SPI, it would mean that SPI listeners are global for all tests. If you want to specify your own annotation type on junit test classes for this specific purpose and specific test, then junit would have to give to a chance to handle class type annotation before calling run(RunNotifier). IMHO the ParentRunner would have to be changed then by adding protected method overridden by yor custom runner handling class type annotation instances.

[pniederw]

@Tibor17 I can't follow your argumentation. My problem is that a @RunWith runner has a "per test class" scope, which means that it can't react to "per test suite" or "per test run" events. I can see at least two ways to support the latter: either by enhancing the @RunWith protocol, or by supporting a JDK service loader (like) mechanism to register a global RunListener. Both of these options should satisfy my needs. PS: I'm not using ParentRunner, I'm implementing a runner from scratch.

[Tibor17]

@pniederw ok, can you describe how you interact with junit framework and what junit classes/interfaces you use in your framework? This is not obvious to me at least. So try to be precise of how much of the code we share .

[pniederw]

For the purpose of this discussion, think of Spock as its own standalone test framework that implements an org.junit.runner.Runner to benefit from the IDE and build tool integration that exists for JUnit. If you are interested in the details, have a look at the implementation of the runner: https://github.com/spockframework/spock/blob/groovy-1.8/spock-core/src/main/java/org/spockframework/runtime/Sputnik.java