Introduce support for parameterized containers (classes, records, etc)

[jkschneider]

Overview

Currently, the target of @ParameterizedTest is constrained to methods. When creating technology compatibility kits, it would be awesome to be able to apply this (or a similar annotation) to the test class so that all tests in that class are parameterized the same way.

Proposal

Rather than:

class MyTest {
   @ParameterizedTest
   @ArgumentSource(...)
   void feature1() { ... }

   @ParameterizedTest
   @ArgumentSource(...)
   void feature2() { ... }
}

Something like:

@ParameterizedTest
@ArgumentSource(...)
class MyTest {
   @Test // ?
   void feature1() { ... }
   
   @Test
   void feature2() { ... }
}

Related Issues

• #14
• #871
• #1141
• #3157

[marcphilipp]

I can imagine supporting something like

@ArgumentSource(...)
class MyTest {
   @ParameterizedTest
   void feature1() { ... }
   
   @ParameterizedTest
   void feature2() { ... }
}

Would that suit your needs?

[jkschneider]

I think that is a good solution.

[nipafx]

Related issues: #871, #853

[jkschneider]

For real-life examples, see GaugeTest and most of the other tests in its package.

[marcphilipp]

@jkschneider Do you have time to work on a PR?

[smoyer64]

If a parameter source annotation is placed on an individual method in a type that's been annotated like this would it make sense for the more specific annotation to take precedence?

[jkschneider]

Yes as a matter of determinism? I think in practice, this would not be the way I'd recommend folks structure their test.

[wrlyonsjr]

Is this feature on the roadmap? I can't migrate to 5 without it.

[wrlyonsjr]

...unless someone knows of a workaround.

[jkschneider]

@wrlyonsjr I flipped it around with Micrometer's TCK by defining an abstract class with my test methods, each of which takes an implementation of MeterRegistry and for which there are many implementations. The RegistryResolver extension injects the implementation into each method at any nesting level.

Then there is a corresponding test for each implementation of MeterRegistry that extends this base class.

The approach has the disadvantage that you can't run the TCK abstract class from the IDE and execute the tests for all implementations, but this is the best I could do.

[wrlyonsjr]

It seems like my problem could be solved with TestTemplate et al.

[sbrannen]

Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.

[dmitry-timofeev]

Let me share my experience with parameterized tests in the new JUnit. Hope that helps to clarify the uses cases and requirements for Parameterized classes.

I think that this feature will be universally useful, not for TCK only. I see its ultimate goal in bringing down the cost of defining multiple tests sharing the same parameters source. Currently, the following code is duplicated:

• Annotations, specifying the source (e.g., @MethodSource).
• Formatting strings for parameters (e.g., "[{index}] = {3}").
• Initialization & Clean-up code, if any. If it has to initialize multiple local variables (i.e., not extractable in a separate method as a whole), it has the highest cost in terms of LOCs.

Having to repeat that code discourages the users to write short, focused tests. On top of that, if developers do have a luxury of copy-pasting that code, reviewers and maintainers have to read all of it.

As an alternative, I've tried a dynamic test for a simple use case (little setup code, no teardown), but got both a personal impression and some reviews from colleagues that it's "overcomplicated".

Uses cases

A perfect use case for this feature, in my opinion, is the following:

1. A user writes a couple of parameterized tests which share the same parameters source and setup code.
2. When there are too many of them, a user extracts these tests in a nested parameterized class (ideally, an IDE inspection tells the user to do that).
   • Test method arguments become either constructor arguments, or injected with @Parameter (as in JUnit 4), or setup method (@BeforeEach) arguments.
   • Initialization code goes to the setup method. Any locals needed in tests become fields of the test class.
   • Clean-up code goes to the teardown method (@AfterEach).

[marcphilipp]

I think supporting something like @Parameter makes sense for fields and method parameters.

I'm out of ideas where you'd put formatting strings for parameters that are shared for all parameterized tests, though.

[dmitry-timofeev]

@marcphilipp , I can think of a class-level annotation with name attribute, or, if users need more flexibility, let them provide an instance method returning a test description + a method-level annotation, or a reference to the method in the class-level annotation (e.g., @Parameterized(name="#testDescription")).

If no one on the core team is planning to work on this issue soon, I may try to implement an MVP. I am new to the code base, and have a couple of questions about the requirements:

• Shall anything except @ParameterizedTest be supported in a class annotated with @ArgumentsSource (@Test, other @TestTemplates)? It might get tricky for one of the most compelling use cases for parameterized classes is extracting test template parameters to the fields, and if there is no extension to resolve the values of these fields, it won't work, will it?
• Shall the extension support a product of primary parameters (defined at the class level) and secondary parameters (defined at the method level, as in the current implementation), like this:

@CsvSource(/* primary parameters, injected into constructor/BeforeEach/fields */)
class FooTest {

  /** Invoked for each parameter in the source specified at the class level */
  @ParameterizedTest
  void foo() { }

  /** 
   * Invoked for the cartesian product of parameters from 
   * the source specified at the class level 
   * and parameters from the source at the method level.
   */
  @ParameterizedTest
  @ValueSource(/* secondary parameters for #bar only */)
  void bar(int secondaryParameter) { }
}

?

[marcphilipp]

I can think of a class-level annotation with name attribute, or, if users need more flexibility, let them provide an instance method returning a test description + a method-level annotation, or a reference to the method in the class-level annotation (e.g., @Parameterized(name="#testDescription")).

Maybe. Let's postpone that part until we've had some more time to ponder it.

Shall anything except @ParameterizedTest be supported in a class annotated with @ArgumentsSource (@Test, other @TestTemplates)? It might get tricky for one of the most compelling use cases for parameterized classes is extracting test template parameters to the fields, and if there is no extension to resolve the values of these fields, it won't work, will it?

How about we annotate fields/method parameters with @Parameter(index=0, optional=true) (optional should be false by default) and inject null for non-parameterized tests?

Shall the extension support a product of primary parameters (defined at the class level) and secondary parameters (defined at the method level, as in the current implementation)

I would expect that the arguments provided by the @ValueSource would be appended to those provided by the @CsvSource in your example, just like when both annotations would be declared on the method.

[sbrannen]

Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.

Actually, the requests I've been hearing are for "parameterized test class" support, not for sharing the same parameters across methods.

In other words, I've been hearing requests to be able to execute all tests within a given test class multiple times with different sets of parameters. When compared to the existing "test template" abstraction, this new feature would rely on a new "test class template" abstraction.

Do people think we can address the issues discussed thus far in this issue with such a "test class template" abstraction?

Or are these orthogonal concerns?

[marcphilipp]

I'm also wondering whether adding something like container templates (cf. #871) would be clearer. If we had those, we could introduce s.th. like @ParameterizedContainer:

@CsvSource(...)
@ParameterizedContainer(name = ...)
class SomeTestClass {

    @Parameter(0)
    private String foo;

    @Test
    void test() {
        ...
    }
}

[sbrannen]

Ahhh.... yes... #871 is what I was looking for!

[sbrannen]

I've added comments to #871 accordingly.

[dmitry-timofeev]

How about we annotate fields/method parameters with @Parameter(index=0, optional=true) (optional should be false by default) and inject null for non-parameterized tests?

Won't that be hard to explain to users and force them to write complicated setup/teardown code, if they have any?

I would expect that the arguments provided by the @ValueSource would be appended to those provided by the @CsvSource in your example, just like when both annotations would be declared on the method.

I see, perhaps, my example failed to convey that I expected different sets of parameters, e.g., @CsvSource({"John, Doe", "Mark, Twain") & @ValueSource(ints = 18, 45).

I think a "test class template" will certainly address the simple use case above, and, likely, clearer, because:

• It will give more flexibility for one can put anything in a "test class template":
  • Regular @Tests that will share both the class-level parameters and the setup/teardown code.
  • Any @TestTemplates, to get a cartesian product of the sets of class parameters, and the parameters of the @TestTemplate, as in example above.
• Resolvers of parameters of a "test class template" and of a "method template" will be independent, will operate on their own level of abstraction, and therefore, be simpler to write and maintain.
• It will yield a hierarchy of tests ("test class templates" -> "regular tests"/"test method templates") that is easier to understand and give names to than a flattened one.

With that much flexibility, "test class templates" are likely to satisfy both simple and advanced or niche uses cases.

[xenoterracide]

I think this might work for my use case #1456, but I'm not certain what the exact syntax would look like reading this issue.

[sbrannen]

but I'm not certain what the exact syntax would look like reading this issue.

Fret not: we also don't know what the syntax will look like. It's up for debate. 😉

[rlundy]

Would it help to have a concrete example to start from?

I'm using JUnit 4 for Selenium tests. I want to run them in multiple browsers. I'm using test class parameterization to do it.

As I have it set up now, a test looks like this:

public class TeamTest extends TestBase {
    public TeamTest(String browser) {
        super(browser);
    }

    @Test
    public void canCreateTeam() {
    // ...

TestBase looks like this:

@RunWith(Parameterized.class)
public abstract class TestBase {

    protected RemoteWebDriver driver;

    protected TestBase(String browser) {
        driver = new DriverCreator().getDriver(browser);
    }

    @Parameters(name = "{0}")
    public static Object[] data() {
        var browserString = Common.getString(Prop.browsers);
        return browserString.split(" *, *");
    }

    // ...

In TestBase, I parse the browser string (which ultimately comes from a configuration file) and then pass that to the test class constructor, which creates browser instances from it.

In each test class, all I have to do is inherit from TestBase and define the constructor. I don't have to decorate individual test methods at all (other than having @Test on them, of course). The test names come out as TeamTest.canCreateTeam[chrome], TeamTest.canCreateTeam[firefox], etc.

So, what might the syntax for this look like if it were added to JUnit 5?

[sbrannen]

So, what might the syntax for this look like if it were added to JUnit 5?

I think it could look very similar to what @marcphilipp proposed here: https://github.com/junit-team/junit5/issues/878#issuecomment-354544841

So, perhaps something like the following:

@ParameterizedContainer(name = "{0}")
@MethodSource("data")
public abstract class TestBase {

    @Parameter(0)
    private String browser;

    protected RemoteWebDriver driver;

    @BeforeEach
    void setUpDriver() {
        this.driver = new DriverCreator().getDriver(browser);
    }

    static Object[] data() {
        var browserString = Common.getString(Prop.browsers);
        return browserString.split(" *, *");
    }

    // ...
}

[lgoldstein]

My 2 cents: I like the flexibility that JUnit 5 provides in this context, but I think that the JUnit 4 parameterized tests paradigm still has a lot of value. The main use-case for it is this: have a test class that accepts the same parameter type/value for all tests. In other words, avoid copy/paste @ParameterizedTest annotation on each method since it is implied from the class-level capability.

@Parameterized
@ValuesSource(...whatever...)
public class FooTest {
    private final Foo testedValue;

    public FooTest(Foo testedValue) {
        this.testedValue = testedValue;
    }

    @Test
    public void testX() { .... }

    @Test
    public void testY() { .... }

    @Test
    public void testZ() { .... }

    ...etc... - no need to repeat @ParameterizedTest on each method -
}

[sbrannen]

@lgoldstein, isn't your @Parameterized proposal identical to the aforementioned @ParameterizedContainer proposal?

[lgoldstein]

Seems like it, a few more thing though

• the @Parameter annotation should be allowed on constructor parameter(s) as well
• If no @Parameter annotation present then behavior should be like JUnit 4 - i.e., vs. implicit positional parameters of the constructor

[marcphilipp]

@lgoldstein Thanks for your 2 cents! 😉

While I haven't made it explicit above, I agree that @ParameterizedContainer should support constructor injection in the same manner we currently support method injection for @ParameterizedTest.

[sbrannen]

@lgoldstein Thanks for your 2 cents! 😉

+1

While I haven't made it explicit above, I agree that @ParameterizedContainer should support constructor injection in the same manner we currently support method injection for @ParameterizedTest.

+1