Spock 2.0 doesn't skip tests that throw org.junit.AssumptionViolatedException

[msgilligan]

Issue description

I've upgraded some Spock tests from Spock 1.3 to Spock 2.0-M3. In a data-driven integration test we are throwing org.junit.AssumptionViolatedException under certain conditions when we can't run the test. In Spock 1.3 this worked correctly, but under Spock 2.0 it shows up as a Failed test.

A clear and concise description. Make sure you choose the correct label as well (bug or enhancement)

How to reproduce

I don't have a simple example that reproduces the problem, but I do have an Open Source project that is failing on Travis CI.

But I think putting the following in a test spec should reproduce the problem:

throw new AssumptionViolatedException("this test should be skipped, not treated as a failure") 

Link to failing Travis CI Job

See https://travis-ci.org/github/OmniLayer/OmniJ/jobs/703291406#L1514

Additional Environment information

Version of your build tool(if used), Java, Groovy, IDE, OS etc

Java/JDK

JDK 8 and JDK 11 (See Travis CI job)

Groovy version

3.0.4 (See Travis CI job)

Build tool version

Gradle

6.5 (See Travis CI job)

Operating System

Ubuntu Xenial (See Travis CI Job)

[msgilligan]

This is (at least tangentially) related to Issue #1008.

[Vampire]

No it is not. :-) org.junit.AssumptionViolatedException is a JUnit 4 class. Spock 2+ is based on JUnit 5. Either throw org.opentest4j.TestAbortedException instead, or add the spock-junit4 artifact to your runtime dependencies as it does this translation amongst other legacy support things.

[msgilligan]

Thanks @Vampire! (Adding the spock-junit4 artifact fixed my problem)

Two suggestions:

1. We add a section to the 2.0 manual for upgrading from Spock 1.3 that suggests adding this dependency
2. Adding a link to the 2.0 manual in README.md

I'm willing to make PRs for both of those if you agree.

[msgilligan]

(for that matter it might be nice to have a landing page somewhere with links to the various versions of the manual)

[Vampire]

Adding the spock-junit4 artifact fixed my problem

Well, it didn't fix it, it work-arounded it. You should better fix it by using the correct exception. The spock-junit4 artifact is for legacy support to help transition to up-to-date stuff and might not be there forever.

We add a section to the 2.0 manual for upgrading from Spock 1.3

That already exists: http://spockframework.org/spock/docs/2.0-M3/migration_guide.html

that suggests adding this dependency

Me, I'm against this. Adding spock-junit4 artifact is a legacy support work-around for the time needed to migrate to proper up-to-date stuff. The release notes at http://spockframework.org/spock/docs/2.0-M3/release_notes.html document as breaking changes that Spock 2+ is no longer based on JUnit 4 but JUnit 5, that using any JUnit 4 elements like their annotations, rules, class rules and exceptions is deprecated and that the spock-junit4 artifact tries to provide legacy support for them on a best effort basis.

Actually you have to have JUnit 4 in your dependencies somewhere, either directly or transitively as dependency of another dependency you use otherwise your tests would not even have compiled, as spock-core does not depend on JUnit 4 anymore since 2.0-M3.

Adding a link to the 2.0 manual in README.md

I would agree to that, but only in the Latest Versions section. The remaining readme is about Spock 1.3. In that section we could add a link to the 2.0-M3 doc and to the 2.0-SNAPSHOT doc.

I'm willing to make PRs for both of those if you agree.

@leonard84 what do you think about the requests?

[leonard84]

1. spock-junit4 should be mentioned in the migration guide, along with a note that this should be seen as a transition aid and not as a permanent solution. @msgilligan if you want to create a PR for that go ahead. Maybe also include a replacement list, e.g., replace org.junit.AssumptionViolatedException with org.opentest4j.TestAbortedException.

2. The problem with the docs is, that there is currently no automation that updates the links. Having a manual entry will be another thing that has to be maintained manually. Having something like the manual/api selection in https://gebish.org/ would be nice, but that would take some work to create the necessary automation. We already have http://spockframework.org/spock/docs/current/ as well as ``http://spockframework.org/spock/docs/latest/`, but they have to be maintained manually. You can see all available versions here https://github.com/spockframework/spock/tree/gh-pages/docs