junit5 icon indicating copy to clipboard operation
junit5 copied to clipboard

Possibility to define implied nodes for hierarchical test engine

Open Vampire opened this issue 1 year ago • 1 comments

While trying to come up with a fix for https://github.com/spockframework/spock/issues/1593, I came to the conclusion that it is probably not possible with manageable effort within the engine, but should be done in JUnit platform, where it imho belongs anyway.

The problem is, the following. Imagine a test class with three test methods a, b, and c. They are configured to must run in order and also to run c you have to run a and b first always, and to run b you have to run a first always. (That's basically the logic of @Stepwise on the test class in Spock)

With Spock 1 which was a JUnit 4 runner this worked perfectly well. But with Spock 2 which is a JUnit platform engine this does not work anymore.

The problem is, that typically (I checked Gradle, Maven, and IntelliJ) a PostDiscoveryFilter is used to match test methods, as the tools usually support some pattern syntax for the method name.

Due to that, the post discovery filter throws out the methods a and b as only c was configured to run. But to correctly run c you have to run a and b first.

I played with different options, but did not yet find a viable solution to solve this properly within the engine, itself. For example using the DynamicTestExecutor given to the execute method of the node for c does not work properly, as then first the setup / "before" method for c is run, then the setup method for b, then for a, then a and b are executed and finally c. This makes problems if you for example work with shared state between the tests. Besides that you then cannot configure needed resources for a or b as those are not supported on dynamically added tests.

So what I'm after and what other test engines might also find useful, is a way to declare / configure that node c implies nodes b and a and that node b implies a. As a result, if after post discovery filtering only c would be left now, it should then also contain a and b. If after post discovery filtering only b would be left, it should also contain a.


An alternative option might maybe be to have some possibility for a PostDiscoveryFilteringPostProcessor, so that the discovery result could again be amended after the post discovery filtering, so that the dependencies can be added back into the tree as needed.


If you have better ideas how to implement it or how I could do it within the engine properly, don't hesitate to tell me. :-)

Vampire avatar Mar 12 '23 19:03 Vampire

In the meantime I came up with an implementation that achieves what we need at https://github.com/spockframework/spock/commit/4c82f7598a5afc235c794442c80e0fef6e8748e2 / https://github.com/spockframework/spock/commit/8214471723180d8753baa507db6ccfe9ab967ebe, where I basically suppress the removeFromHierarchy call, or rather delay it until also all implying nodes for that node were removed and only then actually remove it with supporting only implications within the same test specification hierarchy.

But I still think such a functionality would probably be better placed within the platform code.

Vampire avatar Mar 22 '23 18:03 Vampire