[test goal] run multiple test suites

[redfish4ktc]

Currently, we can only run

• all suites
• a single suite (with the testSuite parameter)

Now, we would like to run more than one suite. This has been requested in the soapui forum (at least in):

• pro: http://www.soapui.org/forum/viewtopic.php?f=3&t=8701
• oss:
  • http://www.soapui.org/forum/viewtopic.php?f=1&t=13725&p=39948
  • http://www.soapui.org/forum/viewtopic.php?f=1&t=14992&p=39951

Possible configuration

<testSuites>
  <testSuite></testSuite>
</testSuites>

If the testSuite parameter is also used, then the test suite will runned as if it has been defined with the testSuites parameter.

[sukoneru]

Hi,

Noticed that this ticket is opened 8 months ago and also is specified in the roadmap(https://github.com/redfish4ktc/maven-soapui-extension-plugin)

Is there any plan to upgrade this plugin with this functionality.

I need exactly this functionality 1- to run multiple tests/testsuites in different projects 2- aggregate the results with the configured junit reporter that is configured with "soapui.junit.reportCollector" property. 3- Also if test in one project fails, then the tests should still contiune to run. As of now, if I configured multiple execution blocks in the plugin, I noticed when one tests in a project fails then the other tests don't get executed.

It did google it and didn't find any solution. But noticed the same suggestions in your roadmap. So wondering if there is anything that's available in near feature.

I will also post it on SOAPUI-PRO forums.

Thanks Surya

[redfish4ktc]

@sukoneru thanks for your feedbacks I currently do not know how to manage multiple projects properly (a lot of people have requested this in the past) as in soapui, the base element to run something is the project not the workspace (ie several projects) This means that we should reimplement a new runner to be able to manage more than one project and also maybe modify the junit collector

To answer to your questions

1. I have plan to superseed the feature describe here by a new one to manage both multiple test suites and test cases. I didn't take time to create the related issue
2. currently no plan for this (see above)
3. see https://github.com/redfish4ktc/maven-soapui-extension-plugin/wiki/TestGoalMultipleProjects, configure the plugin to ignore failures + use the test-verify goal. I created this goal exactly for this purpose. Tell me if the doc is not clear, I will be happy to get your feedback to make it more understandable

I am currently busy on finalizing the new mock-as-war goal and cut a new release. But after that, I could start working on the feature you are interested in (and create the new issue for managing several testsuites/testcases). I previously did this for a user that needed a way to override testsuite properties (see #38)

[tdrury]

FYI: when you do get around to adding this feature, you might want to consider running each soapUI project in it's own process. Our internal soapui maven plugin would eventually use up all the heap after running about 5 projects. So we added a "fork" parameter to have the plugin spawn a new process for each run of our test runner.

[sukoneru]

@redfish4ktc,Thanks a lot for quick reply.

It's really nice to have this plugin.

1 - It will be great if you can work on this piece. But I want to stress the point that we should be able to run multiple test suites and test cases from different projects not just within one project. 2- If you support 1, then will the junit reporter that is configured with "soapui.junit.reportCollector" property display the aggregated results of all the test cases(defined in different projects) that are executed? 3- I will check this and will get back to you.

Can you please let me know the new issue number once you created it. So I will watch it. Once the fix is available, I will plug it.

Thanks again.

[tdrury]

As for aggregating junit test output, we took the "quick and dirty" route. Before letting the runner execute a soapUI project, we set the output folder to some base directory +"/"+ project name. This puts all the output for each project into it's own folder. Then, in Jenkins, we setup the "Publish Junit test result report" section to simply search all of target/soapui-reports/*/TEST-.xml, which then aggregates the reports. But you cannot look at the details of each report without going down into the workspace. That's the sacrifice for quick and dirty.

[redfish4ktc]

Hi guys, sorry for the delay,

@tdrury thanks for the feedback about running several projects. I've just created an issue for this (#88). If you want to provide a pull request or information about what you have implemented, that would be great

@sukoneru

1. I will create the issue soon. I will first work on the OSS runner (because it is open source, and I currently do not know how to implement it with the closed source PRO runner). I guess your need is for the PRO runner, isn't it?
2. as I previously said, I do not plan to manage the aggregation in the runner. This is really specific to each implementation. For example, I like the solution described by @tdrury as the projects are supposed to be independant. The aggregation is delegated to another tool. I know that in your case (regarding what I've seen about your post in the soapui forum), this is not what you expect because you want to generate a single file. Maybe you should create a specific issue for aggregation where we would discuss about this subject. I currently do not see any solution to achieve this.
3. see also #72, instead of using the test-verify goal, delegate the failures detection to jenkins

[redfish4ktc]

@sukoneru #88 has been implemented and is now part of the plugin

[sukoneru]

Thanks a lot. Will check out the new functionality.

On Tue, Apr 15, 2014 at 12:37 PM, Thomas Bouffard [email protected]:

@sukoneru https://github.com/sukoneru #88https://github.com/redfish4ktc/maven-soapui-extension-plugin/issues/88has been implemented and is now part of the plugin

— Reply to this email directly or view it on GitHubhttps://github.com/redfish4ktc/maven-soapui-extension-plugin/issues/41#issuecomment-40524513 .