Support for iOS on BrowserStack

[garyb-bs]

Use case

Currently, Patrol is only supporting Android as part of its BrowserStack integration. We are seeing an uptick in interest for running iOS tests on BrowserStack via Patrol.

Proposal

We want to be able to upload an IPA file to BrowserStack, and upload our Flutter tests all using Patrol.

The Android integration is working well, but without iOS it is hard to give customers what they need for an overall mobile testing experience.

Looking at the bs-android script, it seems the Android integration is using our Espresso URL in the curl command, so it stands to reason that an iOS implementation would utilise the XCUITest URL. The main issue right now is that Patrol seems to use .app files, and BrowserStack is requiring .ipa files. If we could add support for this, it might just work as standard.

[mateuszwojtczak]

Hi @garyb-bs! Thanks for opening the issue. We are using BrowserStack with Patrol in some project and basically the test execution works okay on iOS, but it has a lot of flaws regarding logs, reports and the dashboard itself. We actually filed a couple issues to BrowserStack about the state of XCUITest API.

Let me know how we can proceed on this, because I wouldn't like to keep this issue open without any actionables but also I wouldn't close it because it's not done-done. :)

[garyb-bs]

Hi @mateuszwojtczak - really appreciate the response. When you say the test execution works ok on iOS, what are the steps to run them? Is it simply following the XCUITest steps in the BrowserStack docs here: https://www.browserstack.com/docs/app-automate/xcuitest/getting-started ?

Would you be able to provide me with the list of faults that you provided to the team at BrowserStack? This will allow me to speak with them internally and see what can be done?

[bartekpacia]

Hi @garyb-bs, it's nice to see BrowserStack taking interest in Patrol. As @mateuszwojtczak wrote above, Patrol does work on BrowserStack App Automate for iOS, but the experience is far from perfect, and often there's not much we can do because the problems are not on our side.

The biggest problem is that BrowserStack's XCUITest API v1 (app-automate/xcuitest/v2/build endpoint) doesn't support tests that are generated dynamically at runtime using XCTestCase.testInvocations (and this is what Patrol depends on on iOS). The workaround is to use BrowserStack's XCUITest API v2 (app-automate/xcuitest/v2/xctestrun-build endpoint), which works but has many problems.

First of all, it's impossible to even run tests on BrowserStack's XCUITest API v2 following the official docs. After trying to upload the test suite to the app-automate/xcuitest/v2/test-suite endpoint, we always face the BROWSERSTACK_XCTESTRUN_FILE_VALIDATION_ERROR. We contacted your support about this and the solution they provided is to remove the following key-value pair from the generated .xctestrun file:

<key>DiagnosticCollectionPolicy</key>
<integer>1</integer>​​​​​​​​

This can be automated with the following command:

plutil -remove 'TestConfigurations.TestTargets.DiagnosticCollectionPolicy' MyTestRunFile.xctestrun

but anyway, this is very strange and not a good first run experience.

But once we applied that workaround, we faced the problem with "Test Details" view on BrowserStack's web UI summary looking worse compared to Android web UI summary. Tests execute correctly, but the web UI:

• Always shows there is 1 test, when in reality there's more.

• When at least 1 test fails, it shows "0/1 PASSED" (build ID)

• When no tests fail, it shows "1/1" PASSED" (build ID)

• "Instrumentation logs" are fine – they show correct number of passed and failed tests. But they're not easy to read.

  Example

  "Test Details" is not very readable:

  

  But there are 2 tests listed in "Instrumentation logs":

  

Furthermore:

• BrowserStack XCUITest API v2 requires use of XC Test Plans, which adds significant churn to setup. Other device farms, such as Firebase Test Lab, do not care about this and just work.

• BrowserStack requires .ipas, which adds significant churn to building the app. It's entirely possible to generate .ipa of Flutter app tested with Patrol and make it work on BrowserStack - that's what we do every day in our pipelines - but it's annoying, and device farms, such as Firebase Test Lab, are fine with .apps.

• Our custom error message is cropped in instrumentation logs and in the resulting .xcresult file. Because of that, we don’t know why test failed. It occurs only on BrowserStack, not locally, not on Firebase Test Lab:

  Normal result (locally, Firebase Test Lab)

  

  Bad result (on BrowserStack)

  

• When you open the iOS system settings app (com.apple.Preferences) or the browser (com.apple.mobilesafari), the instrumentations seems to crash but there aren’t error logs and automation run until timeout. This behavior is not documented anywhere.

We look forward to working with you on improving Patrol experience on BrowserStack.

[fylyppo]

Hi @garyb-bs, if you want to try iOS runs by yourself check out our bs_android and bs_ios scripts. Read comments at their top to learn how to use them. If you have any further questions about scheduling runs let us know.

[PiotrWpl]

Hi @garyb-bs ,

Have you had a chance to look more deeply into those issues with integration between Patrol and BrowserStack? My team, we are also very keen to use Patrol together with BS App Automate as we are already using BS App Live, so it would be great to be able to use one platform for both processes.