Provide information about parameterized test iteration in related descriptor

[toolateitsme]

Hello, everyone. Hope, ill find a solution, but it seems, that current PostDiscoveryFilter implementation does not provide filtering parameterized test's iteration

Goal:

Its said, that PostDiscoveryFilter is used for customisation of test tree

I want to exclude (not just skip, like ExecutionCondition) an iteration of parameterized test. Contract of filter says that i need to manipulate on incoming descriptor

Here is a test:

class SomeTest {
    
  @ParameterizedTest(name = "ohh my gosh")
      @ValueSource(ints = [1,2,3])
      fun params(value: Int) {
          assert(value == 1)
      }
}

So, there are 2 failed iterations with unique ID's

[engine:junit-jupiter]/[class:org.example.SomeTest]/[test-template:params(int)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:org.example.SomeTest]/[test-template:params(int)]/[test-template-invocation:#3]

I want to implement a Filter, that will work with some conditions and disable tests, that not matches these unique ID's

But PostDiscoveryFilter receives TestTemplateTestDescriptor instance only with ID, that does not contain unique ID's of iterations above: [engine:junit-jupiter]/[class:org.example.SomeTest]/[test-template:params(int)]

Please, explain me, or add a better context of PostDiscoveryFilter's process.

Can we implement "smart" descriptors for test-template/parameterized tests?

My Environment

• kotlin 1.6 (with JDK 17)
• JUnit5 5.8.2

[marcphilipp]

The invocations are created dynamically at runtime and cannot be filtered with a PostDiscoveryFilter at the moment. Could you implement an ExecutionCondition that skips them instead?

[toolateitsme]

Yep, of course, i can implement this solution.

I've just wanted to make an "clear" test report without "redundant" skipped entries

[toolateitsme]

I was inspired by https://github.com/gradle/gradle/issues/19897 . Its said, that i can provide some rule, and PostDiscoveryFilter will work, imho

[toolateitsme]

@sbrannen Any thoughts? I can close this issue, if we will stuck

[drekbour]

I've tried this and neither PostDiscoveryFilter nor ExecutionCondition get called per dynamic-test. The new IterationSelector https://github.com/junit-team/junit5/pull/2743 is not suitable in this usecase (or mine).

Our usecases both entail a list of uniqueId to allow/block as we choose. The key aspect here is that uniqueId is opaque. We previously observed some uniqueId and in future want to say yes/no to it - regardless of how the test is generated. Least-Surprise says this should be on an existing interface for filtering tests... (I'd much rather PostDiscoveryFilter because I'm trying to filter a million tests so the earlier the better!)

[drekbour]

Ps. I could not register ExecutionCondition via ServiceLoader as I could with PostDiscoveryFilter. Is that not allowed (it is an Extension so I expected it to work)?

[toolateitsme]

@drekbour I Know a solution how to start iteration via ExecutionCondition

And when you have to autoload some JUnit5 features (Extensions, Listeners) - you have to declare a file with a name equal to FQDN package

ex: when you need to autoload some custom Extension - just create a file in resources/META-INF/services with name org.junit.jupiter.api.extension.Extension , then declare a FQDN of your Extension in this file (com.acme.MyCoolExtension as example :))

and dont forget to switch on an autoload feature in build tool

    systemProperty("junit.jupiter.extensions.autodetection.enabled", true)

[drekbour]

Thankyou for the help on that side issue. Now we know more about this, could you rename your Issue to something more direct like 'Ability to filter dynamic tests with ExecutionCondition or PostDiscoveryFilter'. we need to recapture the attention of the devs who probably think IterationSelector is the answer to this one and have stopped looking at it :)

[marcphilipp]

Our usecases both entail a list of uniqueId to allow/block as we choose. The key aspect here is that uniqueId is opaque. We previously observed some uniqueId and in future want to say yes/no to it - regardless of how the test is generated.

@drekbour Have you tried using UniqueIdSelectors for this?

[drekbour]

Not got a computer near to test but will that Selector allow exclusion. My usecase is retaining a small list of good UniqueId from a large data-driven set of generated tests. I want to exclude others by omission . GP wants an explicit deny list (probably of failed tests).

[stale[bot]]

If you would like us to be able to process this issue, please provide the requested information. If the information is not provided within the next 3 weeks, we will be unable to proceed and this issue will be closed.

[marcphilipp]

Team decision: ExecutionCondition is called per test template invocation (i.e. for each repeated/parameterized test invocation) and for @TestFactory methods but not for the returned dynamic containers and tests. Dynamic tests/containers are designed to be light-weight and therefore don't provide any support for extensions but you could evaluate a condition programmatically in the body of the @TestFactory method.