Enhance support for dynamic tests

[sbrannen]

Status Quo

Since 5.0 M1, dynamic tests can be registered as lambda expressions, but there are a few limitations with the current feature set.

Topics

1. Lifecycle callbacks and extensions are not applied around the invocation of a dynamic test.
2. A dynamic test cannot directly benefit from parameter resolution.
   • A dynamic test cannot make use of a TestReporter.

Related Issues

• #14 ~~"Introduce support for parameterized tests"~~
• #386 ~~"Improve documentation of DynamicTest lifecycle"~~
• #393 ~~"TestReporter does not capture the correct TestIdentifier when used with DynamicTests"~~
• #431 "Introduce mechanism for terminating Dynamic Tests early"
• #694 (duplicates this issue, with example)

Deliverables

Address each topic.

[mkobit]

I think it could also be useful/interesting for extensions in this case to do be able to compute "test matrix" by using multiple different parameter resolver extensions and different declarative parameterized values.

For example:

@Test
@ForAValue({3, 4})
@ForBValue({5, 6})
void runMyTest(AClass aValue, BClass bValue) { ... }

I would like to have the ability to have an extension that can register additional tests by looking at the context of a method or class.

[smoyer64]

@mkobit Take a look at PR #577 and the associated proposal in issue #354 ... the idea is to be able to alter the test plan before it's made immutable.

[nipafx]

Regarding lifecycle integration for dynamic tests... What exactly should that look like? Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

Mmmh, now that I'm writing this I don't actually see a problem with that anymore. But I'm sure I saw one when creating #530... Damn brain! :tired_face:

[sbrannen]

Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

That's certainly what I have in mind!

[ledoyen]

Here is a proposition for lifecycle hooks invocations around dynamic tests : #735

[sormuras]

Related issues updated.

[paulmiddelkoop]

At the moment my team is using JUnit 4 in all our tests except when we we need data driven tests. We use Spock there and were very happy with table based testing using where. Writing Groovy tests inside a Java project worked well for us. But since we switched to Kotlin this does not work very well (for instance using Kotlin data classes in Groovy is kind of weak).

So we switched to JUnit 5 and see if we could use the experimental parameterized testing functionality. Personally I don't like if the parameterized data of your test is hidden away in a separate method, class or CSV file. I think one the powers of Spock is that the table is part of the test method. Therefor I really liked the dynamic test approach where we can specify the test data on top of the test and put the real test in an lambda below. Only thing I'm missing here is integration with lifecycles. I think it would be very nice if every single Test in a Dynamic test will have the same lifecycle as a regular @Test. This will also make it easier to use other frameworks like Mockito in dynamic tests.

[paulmiddelkoop]

I don't see why @ParameterizedTest do support lifecycles and dynamic tests don't. For me it's the same concept but in a different form.

[sormuras]

@paulmiddelkoop Please read the comments in https://github.com/junit-team/junit5/pull/735 and https://github.com/junit-team/junit5/issues/371 -- dynamic tests are not real tests. They are more like testlets or a visible form of the soft-assert assertAll(...) method call with its lambdas.

Perhaps you may utilize @TestTemplate or get used to @ParameterizedTest -- both take part in the full life-cycle of real tests.

[mmichaelis]

We just stumbled across this issue because we need the test-lifecycle for dynamic tests. We also thought about using test templates instead, but we need the test instance variables (here wired via Spring) to actually retrieve the test parameters. So our factory method looks something like this:

@ExtendWith(SpringExtension.class)
class SpringTest {
  @Autowired
  private List<Bean> beans;

  @ExtendWith(MyExtension.class)
  @TestFactory
  Stream<DynamicTest> cmTeasableTests() {
    return beans.stream().map(b -> dynamicTest("display name", () -> doTestFor(b)));
  }
}

So unless we find a solution how to access instance level fields from test-templates the test factory is much more convenient to use - but misses the important test lifecycle.

[sormuras]

Why don't you implement doTestFor(b) in such a way that it calls custom before/after methods?

[mmichaelis]

This is what I do as workaround now. But it does not fit to the extension mechanism. So what I do now is to manually create for example after each and handleTestExecutionException by using try-catch-finally. But my impression is, that this is not the expected behavior. It took my a bunch of time to realize that my tests failed, because the cleanup (in after each) was only done when all dynamic tests are done.

[mibutec]

For me Dynamic Tests are handled far to stepmotherly. I'd love to:

• have Lifecycle support for them (as already discussed in this Thread)
• have some kind of usable Surefire feedback for them (see #990)
• discover them early (see #1338)
• add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

[sormuras]

Dynamic tests are just grouped assertions that show up in the test plan. Think of them as assertAll(...) calls on steroids! Nothing more, nothing less. I like to call them "Testlets" on my mind. Dynamic tests are good as they are. They fill a small gap that grouped assertions have: you see all assertions and you can pick a single one an re-run only that ... testlet.

Most of your requested features are already provided by normal or parameterized tests. If those two don't fit your need maybe we can extend them? Perhaps you want and need to roll your own TestTemplate implementation?

[sbrannen]

@mmichaelis,

So unless we find a solution how to access instance level fields from test-templates

What's keeping you from doing that?

Are you just saying you want to access Spring beans from within a custom test template implementation?

If so, that's already possible.

[sbrannen]

add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

That's unfortunately not possible in Java with Java's standard reflection APIs, and any attempt to support such annotation lookups would rely on lambda expression implementation details that may change in future JDK releases.

[sbrannen]

However, if you're curious about some hacky way to achieve that, feel free to take a look at my serialized lambda PoC here: https://github.com/sbrannen/junit-lambda-playground/commit/ef38ddea56ef21771aafcd2b3520f22ee5527973

[sbrannen]

I am sad to say that the "serialized lambda" technique no longer works. See the following JDK issues for details.

• https://bugs.openjdk.java.net/browse/JDK-8138729
• https://bugs.openjdk.java.net/browse/JDK-8140279

It is therefore not possible to perform parameter resolution based on annotations.

Thanks to @nicolaiparlog for bringing this to my attention.

[sbrannen]

we need the test instance variables (here wired via Spring) to actually retrieve the test parameters.

@mmichaelis, this answer I posted on StackOverflow shows how to parameterize tests from Spring beans: https://stackoverflow.com/a/56769619/388980

If you go that route, you have proper test lifecycle method support for each parameterized test invocation.

[ST-DDT]

Would it be possible to just "redefine" DynamicTests / @TestFactory as:

// Virtual part start
@ParameterizedTest
@MethodSource
void testDynamicStuff(DynamicTest test) {
    test.getExecutable().execute();
}
// Virtual part end

// @TestFactory
Stream<DynamicTest> testDynamicStuff() {
      return ...;
}

or at least a new annotation that behaves like this?

[marcphilipp]

Not really since dynamic tests can be nested.

[arun-mano]

Hi Tibor,

I was working with JUnit 5, creating dynamic tests, when I run the build using maven, The XML file under /surefire-reports has the information as expected.

But in the Console logs still lists with indexes, and the test case Displayname is not printed.

I have the below Plugin config:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit-jupiter.version}</version>
        </dependency>
        <dependency>
            <groupId>org.junit.vintage</groupId>
            <artifactId>junit-vintage-engine</artifactId>
            <version>${junit-vintage.version}</version>
        </dependency>
    </dependencies>
    <configuration>
        <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
            <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
            <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
        </statelessTestsetReporter>
        <consoleOutputReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5ConsoleOutputReporter">
            <usePhrasedFileName>false</usePhrasedFileName>
        </consoleOutputReporter>
        <statelessTestsetInfoReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5StatelessTestsetInfoReporter">
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedClassNameInRunning>true</usePhrasedClassNameInRunning>
            <usePhrasedClassNameInTestCaseSummary>true</usePhrasedClassNameInTestCaseSummary>
        </statelessTestsetInfoReporter>
    </configuration>
</plugin>

Error trace:
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.203 s - in null
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] com..........TestCase.testGenericClientsFactory
[INFO] Run 1: PASS
[INFO] Run 2: PASS
[INFO] Run 3: PASS
[INFO] Run 4: PASS
[INFO] Run 5: PASS
[INFO] Run 6: PASS
[INFO] Run 7: PASS
[INFO] Run 8: PASS
[INFO] Run 9: PASS
[INFO] Run 10: PASS
[INFO] Run 11: PASS
[INFO] Run 12: PASS
[INFO] Run 13: PASS
[INFO] Run 14: PASS
[INFO] Run 15: PASS
[ERROR] Run 16: TestCase.lambda$null$3:81->testClientExecute:133->lambda$testClientExecute$7:162 » JSON
[INFO] Run 17: PASS
[INFO] Run 18: PASS
[INFO]
[INFO]
[ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0

Please help me out on this scenario

[marcphilipp]

@arun-mano Please report this issue to the Maven Surefire project.

[t1]

I wrote an extension that supports @TestFactory tests that read tests from files. But I also need setup/cleanup code to run before/after those individual tests, not the whole factory, so I wrote my own @Before/AfterDynamicTest annotations. This is a very limited lifecycle that I think is consistent with the testlet idea described above.

Maybe it would be an idea to leave the current lifecycle as it is and add these annotations instead?

[sbrannen]

Maybe it would be an idea to leave the current lifecycle as it is and add these annotations instead?

That's another option; however, it would not allow individual dynamic tests to benefit from extensions that provide additional behavior via the standard lifecycle callbacks.

[t1]

it would not allow individual dynamic tests to benefit from extensions that provide additional behavior via the standard lifecycle callbacks.

Yes. So dynamic tests are either 'testlets' or have a fully fledged test lifecycles. I'm fine with both options.

[Druckles]

As dynamic tests are functional, would a functional approach be more suitable and more flexible instead of trying to hook into the existing lifecycle & annotations?

E.g.

return DynamicTest.dynamicTest("Test getting users", () -> {
  var result = db.getUsers():
  assertThat(...);
})
  .before(this::setUpDatabase)
  .after(this::tearDownDatabase);

This has the flexibility of being arbitrarily applicable, e.g. to a DynamicContainer:

return DynamicTest.dynamicContainer("Even numbers in database", DynamicTest.stream(...))
  .before(this::setUpForEvenNumbers)
  .after(this::tearDownForEvenNumbers);

This is especially useful if something needs to be performed before or after a group of tests, rather than all tests or each test. I've not found a way to do this other than counting down how many tests have been executed and executing some code when we reach 0.

It can also work to an arbitrary depth of nesting.

[ST-DDT]

I like that approach, but for the container it must be clear, whether the before will be beforeAll or beforeEach (same for after).

[Druckles]

Having thought about it, I would argue that one doesn't need the concept of all/each in this context. It is always before or after whatever container/node you added it to.

This is equivalent to a beforeAll, but the term doesn't make sense semantically on a single node. I.e.

DynamicTest.dynamicTest("Test my function", () -> {})
  .beforeAll(...)

is misleading (as there's only one test).

And to do the equivalent of a @BeforeEach, you can:

DynamicTest.stream(...)
  .map(test -> test.before(...))

[ST-DDT]

@Druckles Just to confirm, is your api call stored in a new field or is it used to mutate the test instance?

• New field => requires usage changes for the test runners.
• Mutates instance => its only syntactic sugar

If it's only syntactic sugar, I don't see a reason, why I should call before manually for all instances/map them myself.