Possibility to run single feature iteration

[Vampire]

As far as I have seen, there is currently no way to run only specific iterations of a feature. So if you run a feature with multiple iterartions and only some fail, you may want to only execute these iterations until you fixed the problem and only after that run all iterations again to see whether you broke something else.

This is also important for IDE support. If you for example in IntelliJ use "Rerun failed tests", or right-click a failed iteration and try to run it, in both cases you are blocked off with "No tests found".

[Vampire]

As the data providers could be one-off iterators, you can of course not construct all the unrolled iteration names up-front to take part in the normal JUnit filtering.

I think we need something like "could the test name in the filter match the unrolled feature name then include the feature" and then when actually arriving at the iteration first check the definitive name against the iteration name and then exclude it at that time.

The first can probably be done like replacing all parts that match UnrollNameProvider.EXPRESSION_PATTERN by .* and then match the unroll-pattern against that. Maybe before replace #featureName by the concrete feature name if it is not followed by a ., so that you do not include all features for something like @Unroll('#featureName #iterationCount') or similar which would result in .* .* otherwise.

[rtretyak]

That would be nice to see in the core spock. Just sharing my experience on how I deal with it now:

1. When debugging a failing iteration one can manually add an assumption on certain iteration markers to the beginning of the test

def "my test"() {
   assumeTrue(iterationNumber == 2)

   expect:
   verify(iterationNumber)
   
   where:
   iterationNumber << [1,2,3]
}

2. Regarding your pattern suggestion. This is currently 100% achievable by cusom extensions (but still a nice feature for core). I do have some examples of including/excluding certain iterations based of generated iteration name. See how we 'tag' certain iterations by their name which afterwards allows us to include/exclude them from the test run https://github.com/telstra/open-kilda/blob/bdbe7d7d0db5dcf8dbac97899a9ec4a35fe2f9d0/services/src/functional-tests/src/test/groovy/org/openkilda/functionaltests/spec/flows/FlowCrudSpec.groovy#L83

[Vampire]

I'm aware of both options, thanks. Actually I tend to just modify the where part, commenting out the iterations I don't want if the iterations need too much time to simply run all, or I want to debug into a specfic iteration.

And the exclusions by extension would of course also work e. g. as I described at https://youtrack.jetbrains.com/issue/IDEA-221964.

But both approaches have significant drawbacks, for example:

• the skipped iterations are still reported as skipped instead of simply suppressed
• it is not the standard way on how to run specific tests
• the standard way is to tell JUnit "I want to run the test named X"

Especially the last point is the most important and tragic one. As you can neither copy the name of a failed test out of the test report and do gradlew test --tests <paste here>, nor can you right-click an iteration in the IDE and run it, or use "Rerun failed tests", as those options depend on the standard JUnit filtering mechanism working like expected.

[l0rinc]

This would be useful for IDE and Gradle runs, e.g. we could assume based on the current method name (with JUnit name rule and an assumption that skips the test if it's not met), like: Spock could probably add these to the setup (or beginning of every test method) and when a target method name or pattern is specified (via variable or System property or whatever), it could ignore every test that doesn't match it.

[leonard84]

With the move to JUnit Platform in Spock 2.0 we have the option to implement this, and it is already on the wanted list. The problem will be to get it to work with idea, since their test matching is currently dependent on the method name, and will not use the ID returned by JUnit Platform IIRC. See https://youtrack.jetbrains.com/issue/IDEA-147105 https://youtrack.jetbrains.com/issue/IDEA-221566

[Vampire]

Those are regarding navigating to source and hopefully one day they adapt to Spock. But if rerunning works properly from IJ, or at least from Gradle, this would already be a great improvement. :-)