spock icon indicating copy to clipboard operation
spock copied to clipboard

Test failures coming from JUnit4 rules are incorrectly attributed to subsequent iterations of unrolled features

Open rieske opened this issue 4 years ago • 1 comments

Issue description

When JUnit4 rule throws while handling an unrolled feature iteration, the test failure is incorrectly attributed to the subsequent iteration. The outcome looks identical to https://github.com/spockframework/spock/issues/1267 and could point to the same root cause.

How to reproduce

import org.junit.Rule
import org.junit.rules.TestRule
import org.junit.runner.Description
import org.junit.runners.model.Statement
import spock.lang.Specification

class RuleSpec extends Specification {

    @Rule
    public SomeRule rule = new SomeRule()

    def "feature with rule"() {
        expect:
        true

        where:
        foo << ['a', 'b', 'c', 'd']
    }
}

class SomeRule implements TestRule {
    @Override
    Statement apply(Statement base, Description description) {
        return new Statement() {
            @Override
            void evaluate() throws Throwable {
                base.evaluate()
                throw new RuntimeException("failure in rule")
            }
        }
    }
}

Executing the above test will report the failures like this: image

Additional Environment information

Build-tool dependencies used

testImplementation("org.spockframework:spock-core:2.0-M4-groovy-3.0")
testImplementation("org.spockframework:spock-junit4:2.0-M4-groovy-3.0")

rieske avatar Jan 29 '21 10:01 rieske

Rule support is made on a best-effort basis, as they don't perfectly match with Spocks concepts, but works well enough most of the time. However, there might be cases - such as this one - where it doesn't perfectly work, I'd suggest to create a native Spock extension if possible.

leonard84 avatar Jan 29 '21 13:01 leonard84