kubeshop
Parallel Testsuite tests execution
Is your feature request related to a problem? Please describe. Currently it's only possible to execute tests concurrently with labels, however it's not convenient and it's not possible to get clear result if all of them passed (as the whole).
Describe the solution you'd like The idea is to allow executing tests in parallel on the Testsuite level. The long term solution may be to allow configuring different Testsuite parts to run in parallel. However, the initial and relatively easy to implement solution would be to allow configuring the whole Testsuite as parallel/concurrent. So if this option will be enabled for specific Testsuite, every Test in the Testsuite will be executed in parallel. Discussed yesterday with @TheBrunoLopes
I think we can go here with really simple implementation:
Just use [][]testStep instead of []testStep.
Current implementation is just a list of testSteps If we would add additional dimension we can create simple matrix e.g.
Before:
- 'Run test 1'
- 'delay 100ms'
- 'Run test 2'
After:
- ['Run test 1', 'Run test 2'] // parallel run
- ['delay 100ms']
- ['Run test 3', 'Run test 2'] // parallel run
cc: @Toxer10 @fivenp @pavloburchak what do you think about UI changes for that? we would need to change the way how frontend behaves (some kind of pipeline definition with quite simple data model behind)
I was about to suggest the exact same array logic for our CRDs @exu :)
Maybe simplyfying it even more by saying whatever is an array is parallel, everything else is concurrent:
- ['Run test 1', 'Run test 2'] // parallel run
- 'delay 100ms'
- ['Run test 3', 'Run test 2'] // parallel run
On the UI this would translate most likely to something similar to this here where you can drag & drop tests around or add new ones (please take it as a concept for now :)):
I would suggest something like this one
In terms of implementation messing with strings and arrays as data model will make it complex so let's keep array (it'll be just one element array which is quite simple to implement on UI)
So, looks like the complete version :smile:
For example, there is a testsuite:
{
"name": "executors-smoke-tests",
"description": "Executors smoke tests",
"steps": [
{"execute": {"name": "cypress-legacy-executor-smoke-electron"}},
{"delay": {"duration": 1000}},
{"execute": {"name": "cypress-10-executor-smoke-electron"}},
{"execute": {"name": "cypress-10-executor-smoke-chrome"}},
{"execute": {"name": "cypress-10-executor-smoke-firefox"}},
{"execute": {"name": "cypress-9-executor-smoke-electron"}},
{"execute": {"name": "cypress-9-executor-smoke-chrome"}},
{"execute": {"name": "cypress-9-executor-smoke-firefox"}},
{"execute": {"name": "cypress-8-executor-smoke-electron"}},
{"execute": {"name": "cypress-8-executor-smoke-chrome"}},
{"execute": {"name": "cypress-8-executor-smoke-firefox"}}
]
}
So, the concurrent steps will look like this:
{"execute": [{"name": "cypress-9-executor-smoke-electron"}, {"name": "cypress-9-executor-smoke-chrome"}]},
?
It was implemented, but frozen by product team for releasing. Ping @TheBrunoLopes for details
hey @michal-kosikcz we are working on making a final polish on it and will be available in the next release.
