ApprovalTests not working with JUnit5's dynamic tests

[juliette-derancourt]

I tried to write the following test using JUnit5's dynamic tests:

import org.approvaltests.Approvals;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;

import java.util.Collection;
import java.util.Collections;

import static org.junit.jupiter.api.DynamicTest.dynamicTest;

public class ApprovalTest {

    @TestFactory
    Collection<DynamicTest> testFactory() {
        return Collections.singletonList(
                dynamicTest("test name", () -> Approvals.verify("result"))
        );
    }

}

It's failing with this error: java.lang.RuntimeException: Could not find Junit/TestNg TestCase you are running, supported frameworks: Junit3, Junit4, Junit5, TestNg

I guess this is the same issue that occured in #36 with @ParameterizedTest. I looked at the related PR (very) quickly and I'm guessing that adding @TestFactory in the list of classes in AttributeStackSelector might solve the problem, unless it's more complicated than it seems 😅.

[LarsEckart]

hello Juliette,

your research regarding #36 and adding TestFactory to AttributeStackSelector looks correct, at least it's also the first thing that I thought when I read the headline. :) I'm currently on vacation and my next planned session with @isidore is on 2nd of August. If you'd like to see this fixed earlier, you can reach out to Llewellyn per email, his address is at the bottom of the readme.md.

[juliette-derancourt]

Hey Lars, thanks for the answer! No worries, it can wait until then 😄 Enjoy your vacation!

[LarsEckart]

It wasn't that easy but we got something. We're about to release 11.8.0, feel free to give it a try and let us know about it. We've tagged it as @Experimental for now. There are a few things you'll need to know to get it to work. We're not gonna mention it here because we're curious if the error message we show makes it self explanatory.

[juliette-derancourt]

@LarsEckart thank you for your work! :)

There are a few things you'll need to know to get it to work. We're not gonna mention it here because we're curious if the error message we show makes it self explanatory.

Good idea :+1:

I was going to test it but I can't find the 11.8.0 release in the maven repository, is it intentional?

[LarsEckart]

Woha, our release-github-action failed and we didn't notice. https://github.com/approvals/ApprovalTests.Java/runs/3225041787?check_suite_focus=true

Sorry about that, we'll get to it next Monday!

[LarsEckart]

We succeeded with the release this time, it's in 12.0.0.

[juliette-derancourt]

Sorry for the late reply, I tested this last week but seem to have forgotten to report back here! :smile:

Your fix works perfectly, and the message is very clear IMO :+1:

However, I'm wondering if wrapping JUnit's API like that is a good idea in the long run? Plus, you might need to add other methods to cover every single way of creating tests (see DynamicTest)

[kqbot3891]

The implementation fails when another class is being used to provide the dynamic tests

public class ApprovalTest {

    @TestFactory
    Collection<DynamicTest> testFactory() {
        return new TestBuilder().getTests();
    }

}

public class TestBuilder {
    public Collection<DynamicTest> getTests() {
        return Collections.singletonList(
            dynamicTest("test name", () -> Approvals.verify("result"))
        );
    }
}

The stacktrace doesn't contain the test class so you're getting a "Could not find Junit/TestNg TestCase you are running, supported frameworks: Junit3, Junit4, Junit5, TestNg"

[LarsEckart]

Thanks for the report and analysis, we'll have another look at it next Monday.