maestro
maestro copied to clipboard
mobile-dev-inc
Sharding/parallel execution improvements
Support for parallel execution has been added in PR #1732, but I see space for more improvement.
Some ideas:
-
[x] #1844 Run in parallel without sharding. In other words, run the whole test on every device.
-
[ ] Allow for some kind of selection what tests can run on what platform.
Related: #1472 If the flow didn't have a key for one of the platforms, it would be skipped on that platform.
-
[ ] Improve parallel/sharded CLI output #1862
Better CLI output, to allow understanding test status at glance.
Maybe it would also be cool if maestro record supported many devices at once?
Request from a user on Slack:
What's the expected workflow to get a single build onto the generated simulators (iOS)?
Right now it has to be done manually with adb install <apk_path>/xcrun simctl install booted <app_path>, but it's obviously not convenient enough.
Maybe Maestro should have a separate install command, and have list of paths to app binaries to install in .maestro/config.yaml? Open for discussion.
Request from a user on Slack:
What's the expected workflow to get a single build onto the generated simulators (iOS)?
Right now it has to be done manually with
adb install <apk_path>/xcrun simctl install booted <app_path>, but it's obviously not convenient enough.Maybe Maestro should have a separate
installcommand, and have list of paths to app binaries to install in.maestro/config.yaml? Open for discussion.
Linked #1665 #1854
When running tests with --shard-split, Maestro tries to evenly split tests between devices. However, there are often tests that run shorter or longer than others. What we end up with is a device that finishes way ahead of the others, and another device that's trying to finish its allocation of tests while the other sit idle.
My suggestion is to have a "shard queue" where all of the tests go into a single queue. When a device (shard) is idle, it pulls the next test from the queue. Perhaps this could be a --shared-queue option if you don't want to change the behavior of --shard-split, but I think this would just be an improvement to --shard-split.
