Failing UI tests can now be retried

[lawmaestro]

• Now using fastlane-plugin-test_center for UI tests - https://github.com/lyndsey-ferguson/fastlane-plugin-test_center
• Added new (optional) environment variable MULTI_SCAN_TRY_COUNT to set the number of times a test is retried before it's considered a fail.
• TAB_UI_TEST_DEVICES now supports the setting of multiple devices, as the name implies. If setting multiple, these should be ':' separated.
• Added new (optional) environment variable MULTI_SCAN_PARALLEL_WORKER_COUNT to set the number of simulators for batched UI tests.
• Added new (optional) environment variable TAB_UNIT_TEST_DEVICE to be able to set a unit test device independently. The device scan parameter was used previously which conflicted with the use of devices by multi-scan.
• Updated changelog for version 6.0.0 changes.
• Updated readme for version 6.0.0 changes.

[KaneCheshire]

Hey! Thanks for taking the time to contribute. This seems like something that could be added locally to your fastfile, rather than complicating the MasterFastfile?

Ideally this would be great to raise as an Issue first so we can have a discussion around it, which also means less wasted time on your part if we decide not to go ahead with these changes.

In a nutshell I'm not sure if we want to add another dependency to our shared MasterFastfile, we use this across multiple projects here at TAB and every dependency needs to be signed off individually by the owners of each project.

[lawmaestro]

Hey @KaneCheshire thanks for the feedback. The reason for adding to the MasterFastfile was just that's where the ui_test lane is. So if we did this all within the local fastfile then in effect we'd be cutting out the MasterFastfile and creating a duplicate ui_test lane.

The motivations behind making this change were as a result of flaky UI tests since migrating to Xcode 11 with parallel UI tests. Therefore I imagined this might be of value across other TAB projects and wasn't too keen on forking etc.

Whether we merge the change or not I'd have still needed to make the same changes, albeit left on a fork so no worries about the time part ;) I'm more than happy to raise an issue for this though. With those points in mind, let me know.

Thanks

[KaneCheshire]

That's kind of the point of the MasterFastfile though, it comes with some basic lanes that anyone can use without many dependencies and then extend it as each project needs individually.

We've been on a bit of a drive to simplify the MasterFastfile recently, I'm not sure this is in line with that really. If it was a case of you couldn't have this functionality at all without it being part of the MasterFastfile (or if something was hugely broken with the current setup) then it would make sense to add it here, but since this is just something you need because you've decided to move to Xcode 11 (not something I'd advise with all the issues!) then it's something your project will just need to handle locally.

Internally at TAB we're advising people not to move to Xcode 11 wherever possible because there's just been too many issues, this is just one example but there's plenty of others like the recent issue with Xcode 11.2 crashing anything older than iOS 13.2 using UITextViews in a storyboard.

Instead I'd recommend filing as many bug reports with Apple as you can (like we are here) and then wait for Apple to address them.

[lawmaestro]

I take your point about the dependencies, I guess on that basis the MS App Center shouldn't be in there either 🤷‍♂️

Technically all functionality is achievable without the Masterfastfile, so on that basis nothing would be in there :) I understand the desire behind simplifying it though.

For us moving over to Xcode 11 isn't really an option as Apple will enforce new submissions be done with 11 in April next year and we've not got capacity early next year to be doing the migration then. Also, waiting on Apple in the hopes they improve the situation with running of parallel tests doesn't feel like a risk we can take. However, the UITextViews issue is addressed already in 11.2.1 :)