Need to abort all test execution on first error

[violinner]

• Is there a way to configure FitNesse to abort all execution of tests, suites, and nested suites on encountering the first error or exception?
• Alternatively, is there a way to add a listener for errors encountered in any test, in any test suite or nested suite?

[fhoeben]

Adding a test listener can be done by calling addTestSystemListener(...) on the test system and TestSystemListener has a method testExceptionOccurred() so you can listen for error there.

https://github.com/unclebob/fitnesse/blob/3b7e2061f72906c1afb0d837550cfbdfd86d5d7a/src/fitnesse/testsystems/slim/SlimTestSystem.java#L102

[violinner]

That would be a change to FitNesse itself, i.e., a new build of fitnesse-standalone.jar, and not accessible through Fixture code, right?

[tcnh]

As always, it depends :) You're using hsac, right? If you're running via the HsacFitnesseRunner, you could extend that runner in your project and override the addTestSystemListeners method. For example:

public class CustomTestRunner extends HsacFitNesseRunner {
    public CustomTestRunner (Class<?> suiteClass) throws InitializationError {
        super(suiteClass);
    }

    @Override
    protected void addTestSystemListeners(RunNotifier notifier, MultipleTestsRunner testRunner, Class<?> suiteClass, DescriptionFactory descriptionFactory) {
        super.addTestSystemListeners(notifier, testRunner, suiteClass, descriptionFactory);
        testRunner.addTestSystemListener(new MyTestSystemListener());
    }
}

[fhoeben]

I haven't tried this, but I believe you can add listeners via plugins (so not via fixtures, but still without changing the fitnesse jar).

I believe you can add a listener by:

• creating it and compiling it into a .jar
• place that .jar the plugins directory
• create (or add a line to) a plugins.properties file containing: Formatters=<your listener class name, inc package>

Instead of having a line the plugins.properties you could also add an implementation of fitnesse.plugins.PluginFeatureFactory and add the listener in its registerFormatters() method. And then add a file META-INF/services/fitnesse.plugins.PluginFeatureFactory to the jar containing the name of the factory class. Examples can be found in hsac-fitnesse-plugin: https://github.com/fhoeben/hsac-fitnesse-plugin/blob/master/src/main/java/nl/hsac/fitnesse/HsacPluginFeatureFactory.java and https://github.com/fhoeben/hsac-fitnesse-plugin/blob/master/src/main/resources/META-INF/services/fitnesse.plugins.PluginFeatureFactory

[violinner]

implementing a fitnesse.plugins.PluginFeatureFactory, which in turn registers a Formatter, does allow a plug-in listener to react to errors and exceptions. All the Formatter methods return void. Throwing a RunTimeException in a plug-in brings FitNesse to an immediate halt with no report of previous successes.

I do not see how that can gracefully tell FitNesse to gracefully skip the rest of the tests and Suites. Is there a way to communicate this from a TestSystemListener, i.e., Formatter, TestsRunnerListener, can do this?

[ALuckyGuy]

If you're using SLIM you can issue a StopSuite exception instead of StopTest. Dosen't that do what you want?

[violinner]

"StopSuite" is correct, but I need to stop the tests within a plug-in.

[violinner]

This was good enough, and works regardless of what version of FitNesse is used:

A test listener plug-in creates a file "fit-stop-test" on encountering an error. (it deletes the file on start-up)

A StartUp .wiki page contains a test table invoking an extended StopTest fixture. On encountering the file "fit-stop-test", StopTest fixture throws a StopSuiteException.

StartUp is inherited for all tests in all the suites, so it causes FitNesse to abort all the Suites after the test listener plug-in "hears about" an error or exception.