sylph
sylph copied to clipboard
mmcc007
Restarting test when devices are available
Is there a solution to restart the test once the devices are available on the Device farm side ? Some time I get the error: Error: device: 'name:Apple iPhone 8 Plus, model:A1864, os:12.1.0, deviceType:ios, formFactor:phone, availability:BUSY' is busy.
If in a CI/CD environment can perhaps auto-retry. Other than that the only options AFAIK are to wait for device to become available or change to a less popular device.
Hi, having a fork here that skips busy devices. Still need to finalize the changes. Also added a random selection of pools. Both is via the sylph.yaml. For us this really helped to get more reliable test executions and good device coverage. Oh and yes, agree with @mmcc007 regarding the rety. We have that as well.
So, summarizing:
- skip busy
- random pool selection
- retry in CI
@mmcc007 Regarding the local dependencies we also added a check to avoid adding packages, which may lead to cyclic dependencies. Would you be interested in a PR?
Happpy Hakking Tim
@tim-fdc Are you able to share your fork, please? Would be incredibly useful!
I gave it a stab: https://github.com/mmcc007/sylph/pull/109
We were facing lots and lots of issues where builds would fail very often as devices were in use. Instead we opted to have a big list of devices to test against, confident this covers the devices in our user base. This means that if any devices were unavailable this was acceptable. But if every device was unavailable that should == a failed step in our CI pipeline.
I've not gone for the full re-try as discussed above. Quite simply because we use GitHub actions and if we waited until a device became available I'd bankrupt the company. Instead, I've set it to skip but it would fail if no devices are available.
Perhaps a further improvement that someone could look into is a toggle on what action you'd like when a device is unavailable. I.e. die if any one device is unavailable / skip unavailable devices and continue with tests.