citrus
citrus copied to clipboard
citrusframework
DataProvider gets invoked many times
Description Assume you have a TestNG DataProvider that returns n parameter sets.
If you execute a parameterized Citrus test / DataProvider, the DataProvider method gets invoked n+1 times.
Citrus Version 2.7.7
Expected behavior The DataProvider method should be invoked exactly once at the begin of the test executions.
Actual behavior The DataProvider method gets invoked once at the begin of the test executions and also gets invoked for every of the n parameter sets (resulting in n+1 executions).
Test case sample
public class TestIT extends TestNGCitrusTestRunner {
@DataProvider(name = "dp")
public static Object[][] dataProvider() {
System.out.println("CALLING DATAPROVIDER");
return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
}
@Parameters({"dataProviderValue"})
@CitrusTest
@Test(dataProvider = "dp")
public void testIT(String dataProviderValue) {
// do nothing
}
}
results in the output:
14:19:38,379 DEBUG citrus.Citrus| Loading Citrus application properties
14:19:38,401 DEBUG BeanDefinitionReader| Loaded 0 bean definitions from location pattern [classpath*:citrus-context.xml]
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,902 INFO port.LoggingReporter| .__ __
14:19:38,902 INFO port.LoggingReporter| ____ |__|/ |________ __ __ ______
14:19:38,902 INFO port.LoggingReporter| _/ ___\| \ __\_ __ \ | \/ ___/
14:19:38,902 INFO port.LoggingReporter| \ \___| || | | | \/ | /\___ \
14:19:38,902 INFO port.LoggingReporter| \___ >__||__| |__| |____//____ >
14:19:38,902 INFO port.LoggingReporter| \/ \/
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| C I T R U S T E S T S 2.7.7
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,903 DEBUG port.LoggingReporter| BEFORE TEST SUITE
14:19:38,903 INFO port.LoggingReporter|
14:19:38,903 INFO port.LoggingReporter|
14:19:38,903 INFO port.LoggingReporter| BEFORE TEST SUITE: SUCCESS
14:19:38,903 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,903 INFO port.LoggingReporter|
CALLING DATAPROVIDER
14:19:38,919 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:38,926 INFO port.LoggingReporter|
14:19:38,926 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,926 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT <test.citrus.DataProviderParallel>
14:19:38,926 INFO port.LoggingReporter|
14:19:38,926 DEBUG citrus.TestCase| Initializing test case
14:19:38,927 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:38,927 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:38,927 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:38,927 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'a'
14:19:38,927 DEBUG citrus.TestCase| Test variables:
14:19:38,927 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:38,927 DEBUG citrus.TestCase| dataProviderValue = a
14:19:38,927 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:39,929 INFO port.LoggingReporter|
14:19:39,930 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:39,930 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:39,930 INFO port.LoggingReporter|
14:19:39,931 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:39,932 INFO port.LoggingReporter|
14:19:39,932 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:39,932 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT <test.citrus.DataProviderParallel>
14:19:39,932 INFO port.LoggingReporter|
14:19:39,932 DEBUG citrus.TestCase| Initializing test case
14:19:39,932 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:39,932 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:39,932 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:39,932 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'b'
14:19:39,932 DEBUG citrus.TestCase| Test variables:
14:19:39,932 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:39,932 DEBUG citrus.TestCase| dataProviderValue = b
14:19:39,932 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:40,932 INFO port.LoggingReporter|
14:19:40,933 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:40,933 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:40,933 INFO port.LoggingReporter|
14:19:40,934 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:40,935 INFO port.LoggingReporter|
14:19:40,935 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:40,935 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT <test.citrus.DataProviderParallel>
14:19:40,935 INFO port.LoggingReporter|
14:19:40,935 DEBUG citrus.TestCase| Initializing test case
14:19:40,935 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:40,935 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:40,935 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:40,936 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'c'
14:19:40,936 DEBUG citrus.TestCase| Test variables:
14:19:40,936 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:40,936 DEBUG citrus.TestCase| dataProviderValue = c
14:19:40,936 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:41,936 INFO port.LoggingReporter|
14:19:41,936 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:41,936 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,936 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 DEBUG port.LoggingReporter| AFTER TEST SUITE
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| AFTER TEST SUITE: SUCCESS
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| CITRUS TEST RESULTS
14:19:41,940 INFO port.LoggingReporter|
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter|
14:19:41,942 INFO port.LoggingReporter| TOTAL: 3
14:19:41,942 DEBUG port.LoggingReporter| SKIPPED: 0 (0.0%)
14:19:41,942 INFO port.LoggingReporter| FAILED: 0 (0.0%)
14:19:41,943 INFO port.LoggingReporter| SUCCESS: 3 (100.0%)
14:19:41,943 INFO port.LoggingReporter|
14:19:41,943 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,943 DEBUG report.HtmlReporter| Generating HTML test report
14:19:41,944 DEBUG util.FileUtils| Reading file resource: 'test-detail.html' (encoding is 'UTF-8')
14:19:41,961 DEBUG util.FileUtils| Reading file resource: 'test-report.html' (encoding is 'UTF-8')
14:19:41,962 INFO ctOutputFileReporter| Generated test report: target/citrus-reports/citrus-test-results.html
14:19:41,963 DEBUG report.JUnitReporter| Generating JUnit test report
14:19:41,964 DEBUG util.FileUtils| Reading file resource: 'junit-test.xml' (encoding is 'UTF-8')
14:19:41,969 DEBUG util.FileUtils| Reading file resource: 'junit-report.xml' (encoding is 'UTF-8')
===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================
Process finished with exit code 0
As you can see the line
CALLING DATAPROVIDER
is present in the output 4 times, while it should be present only once.
Note To make sure that this is actually a Citrus issue I created the same test without Citrus / only with TestNG - without Citrus the DataProvider method gets invoked only once (expected behavior).
public class TestWithoutCitrusIT {
@DataProvider(name = "dp")
public static Object[][] dataProvider() {
System.out.println("CALLING DATAPROVIDER");
return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
}
@Parameters({"dataProviderValue"})
@Test(dataProvider = "dp")
public void testIT(String dataProviderValue) {
// print value to make sure that the test gets invoked n times
System.out.println("DataProviderValue " + dataProviderValue);
}
}
results in the output
CALLING DATAPROVIDER
DataProviderValue a
DataProviderValue b
DataProviderValue c
===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================
Process finished with exit code 0
As you can see the line
CALLING DATAPROVIDER
is present in the output only once, which is the expected behavior.
Hi!
Thx again for another detailed issue!
Yeah, this lies in the details of the TestNG integration with Citrus. Have you experienced functional deficiencies based on these multiple invocations?
BR, Sven
Hi!
Sorry for my late reply.
No, currently I did not expect any functional deficiencies.
I use the dataprovider to create a huge amount of different parameter sets. The overall test execution time is negatively influenced by this issue, because it takes quite some time for the dataprovider to create those sets not only once, but n+1 times.
Our current workaround is to create a private static Object[][] dpResult = null; field that is filled and returned by the dataprovider during the first invocation / that is returned by the dataprovider on all following invocations.
Hi!
Okay, now I got your point. Yeah this is definitely worth a fix.
BR, Sven
Hi,
I faced same issue , but I have functional deficiencies because I'm using dataprovider to read from excel and parametrize my test, therefore read from excel function is invoked multiple time, it leads to resource wasting/locking.
Would be fixed this issue? Note: I'm using Citrus version 3.0.0-M3
Thanks, Vitalie
Sadly this limitation appears to have existed for years. It seems this should be a higher priority as anyone attempting to use TestNG would not use this in a real-world automation solution.
