junit4

junit4 copied to clipboard

Theories test runner is powerful but has some drawbacks:

• When error happen, it's hard to know what parameter combination is making that error
• Do not have option to continue the test method with other parameter combination if one failed. Currently for a test method, it stops if failed with any parameter combination.

Sorry for the long silent period on this feature request. There are two different requests here. Can you split them into two github issues? Thanks.

It looks really interesting - I will try to tackle them at friday

It looks like that second feature is implemented:. I mean running Theory with multiple possible parameters results in running all posible combination of parameters. What is also really important given "theory" fails only if all possible sets of parameters fails. As a result failing parameters will be printed if all combination of parameter cause test failure. I suggest to add field for Theory annotation:

• printIncorrectParameters of type boolean - result in always printing incorrect cases (name of parameter for discussion)

In my opinion it will be better if test fails in situation in which at least one combination of parameters result in failure. If you do not agree with this statement I suggest to make it configurable I mean add some field for theory annotation that will decide whether test fails if all of possible parameters result in failure or at least one set of parameters results in failure. I'm waiting for comments

Hi,

I think that the parameters that caused the test do fail should be inside of the error message, so the it would be easy to find the cause of the test failure. Is that possible? I also think that this should be the default behavior, the developer always want to knwow why the test failed.

What do you think? I can try to implement this and start contributing with the project.

@csokol The parameter that caused the test to fail is already visible in the error message.

@RunWith(Theories.class)
public class TheoryTest {

    @DataPoints
    public static String[] datapoints = { "a", "b", "wrong" };

    @Theory
    public void myTheory(String parameter) throws Exception {
        assertThat(parameter.length(), is(1));
    }
}

results in

org.junit.experimental.theories.internal.ParameterizedAssertionError: myTheory(datapoints[2])
    at org.junit.experimental.theories.Theories$TheoryAnchor.reportParameterizedError(Theories.java:192)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1$1.evaluate(Theories.java:146)
    at org.junit.experimental.theories.Theories$TheoryAnchor.runWithCompleteAssignment(Theories.java:127)
    at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:111)
    at org.junit.experimental.theories.Theories$TheoryAnchor.runWithIncompleteAssignment(Theories.java:120)
    at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:109)
    at org.junit.experimental.theories.Theories$TheoryAnchor.evaluate(Theories.java:96)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.AssertionError: 
Expected: is <1>
     but: was <5>
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
    at org.junit.Assert.assertThat(Assert.java:865)
    at org.junit.Assert.assertThat(Assert.java:832)
    at TheoryTest.myTheory(TheoryTest.java:17)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.experimental.theories.Theories$TheoryAnchor$2.evaluate(Theories.java:175)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1$1.evaluate(Theories.java:141)
    ... 20 more

You're right, @marcphilipp. But notice that is still hard to read it, and if the test had failed for some other reason than an assertion in the parameter value, it would not be shown. I think that the first exception message should show the parameter value that failed the test. What do you think?

What other reason? It will be shown whenever an exception is thrown from somewhere inside the theory method. The theory will fail if and only if an exception is thrown from the theory method. Thus, whenever the theory fails the exception will be wrapped in a ParameterizedAssertionError.

I partly agree with your argument that the complete stacktrace is hard to read. However, most IDEs will filter the stacktrace and only show the relevant part. E.g. in Eclipse this looks like:

To me, this looks fairly clear.

If I have something like this:

    @DataPoints
    public static double[] datapoints = { 2.0, 4.0 };

    @Theory
    public void myTheory(double d) throws Exception {
        double pow = BogousMath.pow(d, 2);
        assertThat(pow, is(d*d));
    }

    static class BogousMath {
        public static double pow(double d, int i) {
            return 0;
        }
    }

The example is stupid, but is a fair usage of theories.

You coulf build a custom String message that contains the value of your parameter and pass it to assertThat. Would that work for you?

It works, but it's it would be easier if the framework did that for us... Is it too hard do implement?

Chico Sokol

On Thu, Jun 26, 2014 at 6:43 PM, Marc Philipp [email protected] wrote:

You coulf build a custom String message that contains the value of your parameter and pass it to assertThat. Would that work for you?

— Reply to this email directly or view it on GitHub https://github.com/junit-team/junit/issues/180#issuecomment-47284040.

It seems obvious for primitive types, agreed. However, it also has to work for arbitrary types where toString() does not always print something short and helpful.