junit5
junit5 copied to clipboard
junit-team
Introduce support for zero invocations in test templates and parameterized tests
Overview
Using: JUnit: 5.2.0
Great job on the new ParameterizedTest support in v.5. The replacement of the static, one-per-class Parameters annotation with more flexible MethodSource, etc. has been like a breath of fresh air and allowed me to remove thousands (!!) of lines of supporting code from my system. I'm really loving it.
However, someone decided to disallow zero parameters with this precondition check in ParameterizedTestExtension.
.onClose(() ->
Preconditions.condition(invocationCount.get() > 0,
"Configuration error: You must provide at least one argument for this @ParameterizedTest"));
The problem with this is that we have some testing situations where parameterized tests with zero parameters are not exceptional. For example, we run tests against thousands of a certain type of class generically against a database of past production failures, and many of these classes have never experienced a production failure. When the tests are run, we now get failures due the above precondition check. JUnit 4 handled this cleanly: it would simply not run any tests on those classes.
Workaround
I can get around this by adding a null to the method creating the collection of parameters if it is empty, and then return from the beginning of the @ParameterizedTest method code if the passed parameter is null. That lets us continue to run the parameterized tests against all of the classes, but comes with some disadvantages:
- We must add specific code to the front and back of every parameterized pair just to avoid a failure that doesn't matter to the tests.
- The handling of the
nullcauses the test count inflation for these "phantom" tests. -
nullis now reserved as a signal for no parameters, rather than something wrong in the parameter creation.
Proposal
If nobody has any strong feelings about disallowing the no-parameter case, can we just have this precondition removed from a future version?
Thanks.
@johnchurchill Could you please provide a few more details about your use case, particularly how you've written your tests? I assume it involves base classes that inherit @ParameterizedTest methods?
I have also been trying to envision what you are trying to do and was not able.
My best guess is that you have some sort of ArgumentsProvider that returns no elements, but still I am not sure. If my guess is valid, this behaviour was explicitly added as a solution for #923.
It would be great if you could provide a skeleton test that exhibits your issue.
Yes, there is a superclass of all tests which does the parameter loading and execution. There are thousands of subclasses that implement all kinds of functionality. Each (non-test) class is the logic behind an interactive web page in the system. Here is a boiled-down version of that superclass:
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SpringDevTestServerConfiguration.class)
@TestExecutionListeners(listeners = { DependencyInjectionTestExecutionListener.class })
@ComponentScan(lazyInit = true)
@TestInstance(PER_CLASS)
public abstract class AbstractConfigTestClass implements ApplicationContextAware {
protected List<SingleConfigTestDescriptor> configTests() {
String testClassName = this.getClass().getName();
// remove "Test" on the end to obtain the config class
String clzName = testClassName.substring(0, testClassName.length() - 4);
try {
Class configClz = Class.forName(clzName);
List<SingleConfigTestDescriptor> params = ConfigToTestParams.getTestParameters(configClz);
if (params.size() == 0) {
params.add(null); // need to add null here to avoid junit 5 failing
}
return params;
}
catch (ClassNotFoundException e1) {
throw new RuntimeException("Could not find expected class: " + clzName, e1);
}
}
@ParameterizedTest
@MethodSource("configTests")
public void runTest(SingleConfigTestDescriptor test) {
if (test == null) {
return; // nothing to do; this is the null to avoid the junit 5 failure
}
// use the test data to run the test against the class
}
}
By the time we know that there are no available tests to run, it's too late.. JUnit 5 will throw an exception. There are 6 lines in there whose only purpose is to make it run successfully like it did under JUnit 4. I just started working with JUnit 5 a couple of days ago, so I haven't had the chance to look into using ArgumentsProvider. Would using ArgumentsProvider avoid the failure?
In any case, we should improve the error message.
"Configuration error: You must provide at least one argument for this @ParameterizedTest" --> "Configuration error: You must provide at least one invocation context for this @ParameterizedTest".
The difference being "argument" vs. "invocation context".
And the invocation context can still supply zero "arguments" to the method, though that's not the topic of this issue.
Keep in mind that a @TestTemplate method in general cannot currently have zero invocations.
In other words, see org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(int). 😉
I took a look at TestTemplateTestDescriptor, and yes the zero invocations would be stopped there too. I tried taking both of those preconditions out (TestTemplateTestDescriptor and ParameterizedTestExtension) and making a simple class with one @Test and one @ParameterizedTest which refers to a @MethodSource which returns an empty list. The result is that 1/1 tests pass (just bar2()), which is the same behavior we had in JUnit 4. There is no side-effect of allowing the @ParameterizedTest to run with no invocations.
public class FooTest {
@ParameterizedTest
@MethodSource("data")
void bar1(String s) {
}
@Test
public void bar2() {
// pass
}
public static List<String> data() {
// pretend we have no test cases in the QA database
return new ArrayList<>();
}
}
There is no side-effect of allowing the
@ParameterizedTestto run with no invocations.
Well, that's only partially true.
It's a breaking change in the sense that configuration errors previously caused the parameterized test container to fail; whereas, now such configuration errors would be silently ignored.
Of course, your use case is not a "configuration error". So we have to decide how to best support both scenarios. 😉
I suppose, if we wanted to support zero invocations, we could simply generate a log message informing the user. The tricky part is what log level to choose. Is that DEBUG, INFO, or WARN?
The result is that 1/1 tests pass
In such cases, I don't think the parameterized test container should be marked as "successful" (or somehow ignored, whatever was the case when you implemented that PoC).
Rather, I think we should mark the parameterized test container as "skipped" and provide a "reason" why it was skipped.
@junit-team/junit-lambda, thoughts?
As a side note, since I'm the author of the Spring TestContext Framework...
@johnchurchill, out of curiosity, why do you disable all Spring test listeners except DependencyInjectionTestExecutionListener?
That's enabled by default.
Were other listeners somehow causing problems?
This case reminds me of the similar case #1298 which was handled by the --fail-if-no-tests cl option.
The similarity is that they both exhibit the same core dilemma: If I have zero elements to process, is that because the user has done some error or is this lack of items actually valid and intentional? The error could be in configuration as in both cases or programmatic in the case of providing arguments_(invocation contexts)_.
The difference between the two is the default is reversed.
- In the case of the Console Launcher. In cl the default was not to fail and we added an option to change this behavior.
- For the parameterized case we fail if zero invocation contexts.
Maybe we could take a similar approach also in the case- with some sort of flag or parameter eg on the annotation.
For me personally I think the current default to fail if no elements are present and in esense guarding against error is important. I seen many case where something was silently turned off due to error and the error was missed. *Also adding a skipped note in the execution plan could be in many cases be effectively silent if it does not actually fail a test run
n any case, we should improve the error message.
"Configuration error: You must provide at least one argument for this @ParameterizedTest" --> "Configuration error: You must provide at least one invocation context for this @ParameterizedTest".
The difference being "argument" vs. "invocation context".
I don't think invocation context would be more meaningful for the normal user. Though indeed it is more meaningful for someone who understands the internal of the platform.
If you consult the relevant section in the User Guide about ParameterizedTests there is no mention, let alone an explanation, of "Invocation Context".
Sure.... "invocation context" is very technical.
How about the following instead?
"Configuration error: You must configure at least one set of arguments for this @ParameterizedTest"
Good points, @gaganis.
Maybe we could take a similar approach also in the case- with some sort of flag or parameter eg on the annotation.
I tend to agree that adding an opt-in flag as a annotation attribute in @ParameterizedTest would be a better, more robust, and more user friendly solution.
How about the following?
@ParameterizedTest(failIfNoArgumentsProvided = false)
... with the default being true.
Though, failIfNoArgumentsProvided is a bit verbose. Perhaps we can come up with something more succinct.
Maybe the following is better:
@ParameterizedTest(requireArguments = false)
How about the following instead?
"Configuration error: You must configure at least one set of arguments for this @ParameterizedTest"
That is exactly was I was thinking now too :)
@johnchurchill, would the following proposal suit your needs?
@ParameterizedTest(requireArguments = false)
FYI: I improved the error message in b1e533cf64bedb7ebbb8c80e9141fbb4cd515d0d.
Yes, requireArguments = false would suit my needs perfectly. I hate to see more corner-case parameters being introduced though. The simple API is what makes JUnit so approachable.
Yes,
requireArguments = falsewould suit my needs perfectly.
Thanks for the feedback.
I hate to see more corner-case parameters being introduced though. The simple API is what makes JUnit so approachable.
Well, therein lies the rub: We can't have our cake and eat it, too.
As already pointed out, switching the behavior would no longer be user friendly regarding user configuration errors. So, IMHO, the only viable option is to make it an opt-in feature while retaining backwards compatibility and upholding UX.
Of course, if anyone has any better ideas, I'm all ears.
@johnchurchill You can just use an assumption in your configTests() method like this to skip the complete @ParameterizedTest:
assumeFalse(params.isEmpty(), "nothing to test");
Indeed, a failed assumption in the factory method is an easy way to achieve this without changes to existing APIs.
Nice idea, @marcphilipp! 👍
Actually, it was @sormuras who had the idea during yesterday’s team call. 😉
Hi, Thanks for the suggestion. I just got to a stopping point where I could try out the assumption approach. However, either I'm misunderstanding where it goes, or it doesn't apply to this situation. In the docs, the assumptions abort a test within a test-annotated method, not within the method that gathers and returns the test parameters. TestAbortedException is thrown, which causes an error. Here is how I am using it, modified from my original code above.
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SpringDevTestServerConfiguration.class)
@ComponentScan(lazyInit = true)
@TestInstance(PER_CLASS)
public abstract class AbstractConfigTestClass implements ApplicationContextAware {
protected List<SingleConfigTestDescriptor> configTests() {
String testClassName = this.getClass().getName();
// remove "Test" on the end to obtain the config class
String clzName = testClassName.substring(0, testClassName.length() - 4);
try {
Class configClz = Class.forName(clzName);
List<SingleConfigTestDescriptor> params = ConfigToTestParams.getTestParameters(configClz);
Assumptions.assumeFalse(params.isEmpty(), "nothing to test");
return params;
}
catch (ClassNotFoundException e1) {
throw new RuntimeException("Could not find expected class: " + clzName, e1);
}
}
@ParameterizedTest
@MethodSource("configTests")
public void runTest(SingleConfigTestDescriptor test) {
// use the test data to run the test against the class
}
}
Result when no test parameters exist:
org.opentest4j.TestAbortedException: Assumption failed: nothing to test
TestAbortedException is thrown, which causes an error.
Can you be more explicit about what error you see (i.e., reported for what)?
Never mind. I simplified the test case as follows in order to reproduce the error locally.
import java.util.List;
import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
class TestCase {
static List<String> strings() {
Assumptions.assumeFalse(true, "nothing to test");
return null;
}
@ParameterizedTest
@MethodSource
void strings(String test) {
}
}
For the above TestCase, I can confirm that a failed assumption results in a failure for the strings(String) parameterized test method as follows when executing against the current master.
Note the suppressed PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this @ParameterizedTest.
org.opentest4j.TestAbortedException: Assumption failed: nothing to test
at org.junit.jupiter.api.Assumptions.throwTestAbortedException(Assumptions.java:255)
at org.junit.jupiter.api.Assumptions.assumeFalse(Assumptions.java:191)
at TestCase.strings(TestCase.java:10)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:515)
at org.junit.jupiter.params.provider.MethodArgumentsProvider.lambda$1(MethodArgumentsProvider.java:46)
// omitted
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:101)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:1)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$4(NodeTestTask.java:131)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:107)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1378)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$4(NodeTestTask.java:135)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:107)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1378)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$4(NodeTestTask.java:135)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:107)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:52)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:184)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$6(DefaultLauncher.java:152)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:166)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:145)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:92)
at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
Suppressed: org.junit.platform.commons.util.PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this @ParameterizedTest
at org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:280)
at org.junit.jupiter.params.ParameterizedTestExtension.lambda$9(ParameterizedTestExtension.java:87)
at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:281)
... 39 more