[Feature] Run Playwright tests on real mobile device browsers?

[Cosmin-Gramada]

trafficstars

Is there any way to connect to a device, start a browser (e.g. chromium instance on Android) and run the tests?

If not, what is missing at this point to be able to use it on both Android and iOS?

---

Note from maintainers:

Please refer to the https://playwright.dev/docs/next/api/class-android for instructions on how to use Playwright with Chrome for Android, Android WebViews and Android itself

---

[pavelfeldman]

There is no way to attach to Chromium on Android or MobileSafari on iOS. Playwright aims at running on your desktop and in your CI/CD, targeting device farms is outside its scope.

We will emulate device metrics, geolocation, orientation, etc on Chromium on Android and MobileSafari though. Could you share more about your use case?

[Cosmin-Gramada]

We took into account the possibility of moving away from Selenium WebDriver based tools and switching to DevTools API based tools.

A really big portion of our tests are running on mobile web browsers, where we tests our own player that makes use of browser related technologies (e.g. what types of medias is able to render, what's the appropriate format, etc) and sends specific tracking events.

Our concern is that there will be differences between a real device and an emulated one, as we're able to emulate any device resolution, but only the viewport size is guaranteed to be accurate.

[pavelfeldman]

Our concern is that there will be differences between a real device and an emulated one

Yes, there will be

as we're able to emulate any device resolution, but only the viewport size is guaranteed to be accurate

More than the viewport size, but yes, if you need mobile-specific media to be accurate, you need to attach to the device.

This is technically feasible, but is outside of the scope for v1.

[tanaypratap]

This could be useful for us too. We are a fully fledged PWA email app. Testing on actual devices would catch bugs. We have also faced this issue with non-existent media queries on some phones which @Cosmin-Gramada mentioned. Especially older phones.

[Filipoliko]

Since Playwright is trying to be more testing-friendly, it would be really great to have real mobile device support. More and more users are coming to our websites from mobile devices, it is nearly 50:50 for us. Most of our bugs on mobile devices are not discoverable via simple mobile emulation. It is really important for us to be able to test on real mobile devices, or at least on emulated Android/iOS systems.

[arjunattam]

@Filipoliko - thank you for sharing. Can you elaborate on which bugs are not discoverable via emulation? Examples will really help!

[Filipoliko]

@arjun27 - They are usually some very specific (and kinda weird) use cases, typically on iOS devices. One current bug on our website, that is discoverable only on real iOS device (safari, or chrome) can be currently found on this page. (The fix will be probably released during next week, hopefully you can check it before that time).

If you click on Domácí (see the picture below), then with chrome mobile emulation, you will be redirected to a section page Domácí (correct behaviour). But if you are running chrome, or safari on real iOS device, then the click will open the mobile menu. This is fixed by setting pointer-events: none on the hidden mobile menu element. Android/Windows/macOS works fine, but iOS requires this css rule for some reason.

Probably most typical use cases (as far as I can think of) are related to some invisible elements overlapping other elements and making it impossible to click on the overlapped element. This behaviour is usually iOS specific, other systems interpret these css rules in a bit different way.

[arjunattam]

Thanks @Filipoliko. I just tried running this case with iPhone emulation on Playwright WebKit and I'm able to repro this issue. Chrome mobile emulation is not reproducing it, as you've rightly pointed it out. Also, you're right about an invisible overlapping element (which is why I had to use force: true to issue the click). Sharing code below in case you'd like to try it out.

const { webkit, devices } = require('playwright');

(async() => {
  const browser = await webkit.launch({headless: false});
  const context = await browser.newContext({...devices['iPhone 11 Pro']});
  const page = await context.newPage();
  await page.goto('https://www.seznamzpravy.cz/clanek/v-korupcni-kauze-stoka-obvinila-policie-dalsi-27-lidi-111820');
  await page.click('[data-dot="ogm-breadcrumb-navigation"] >> text=Domácí', {force: true});
})();

[Kradirhamik]

Hi,

I would like to also give support for this idea. Our use cases requires our Angular site to get input from other apps (e.g. BankID for authentication). So the real flow can only be properly tested on a real device.

[alejandroiglesias]

Hi, I'm curious to know what the workflow is on this project for an issue like this to move from feature request to development by the team? Does it need to reach a certain amount of upvotes? Does it need any funding? Does it take a certain amount of time to pass, like waiting for the next major release planning? It would be good for users to know and be able to set expectations on the development of feature requests. Thanks!

[pavelfeldman]

We have already seen enough interest in it, so we will invest into this area. There are several questions about the use cases and the API, so we would appreciate your feedback, the more details the better. I'll post some initial questions below as comments, please upvote them in case they apply!

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright with the Chrome Browser (WebView integration is not essential for you).

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright against your WebView and can bring WebView into a testable state over adb.

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright against AVD (Emulator) as opposed to the real device.

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright against the physical device as opposed to AVD (Emulator).

[pavelfeldman]

Please 👍 if

Your testing setup requires Android w/ Play Store

[pavelfeldman]

Please 👍 if

Your testing setup does not require Android w/ Play Store

[pavelfeldman]

Please 👍 if

You intend to run tests in the cloud.

[pavelfeldman]

See https://playwright.dev/docs/api/class-android for experimental support.

[pavelfeldman]

~~Folks, npm i playwright-android is now live and it has a first cut of the support for Android's Chrome browser as well as WebViews. It has not been covered with the tests, neither it runs on the bots. But it does work for simple scenarios. Please let us know if it works for you or if something is missing. As a drive-by it includes uiautomator-alike driver for the Android itself.~~

Edit: Please reference to the guide here: https://playwright.dev/docs/api/class-android

[alejandroiglesias]

@pavelfeldman that's great! Thanks for the initial implementation efforts. Are there any plans to do the same for iOS Safari? That would be awesome to get the main mobile contenders covered. Also, do you know how/if this could work with @web/test-runner-browserstack (a test runner that supports Playwright)?

[vania-pooh]

@alejandroiglesias so far as I know BrowserStack now supports Selenium protocol only.

[alejandroiglesias]

@vania-pooh BrowserStack also supports Cypress and has an HTTP-based API, and if @web/test-runner-browserstack does what it promises and can be combined with @web/test-runner-playwright, I would imagine that Playwright it should work in BrowserStack... in theory and unless I'm missing something. Alternatively, SauceLabs supports Playwright as it's stated in their docs.

[ydennisy]

@vania-pooh is the browserstack integration something you would consider?

[vania-pooh]

@ydennisy we personally don't need this.

[javiercn]

Does this feature item cover the new Edge WebView2 control for Windows? Is there a way to make that work with the current bits?

[kaivean]

@pavelfeldman Hi, Referring to https://github.com/microsoft/playwright/blob/v1.9.1/docs/src/mobile.md , I find the npm package lack bin/android-driver-target.apk and bin/android-driver, so coping them to the directory node_modules/playwright/bin. Finally I successfully launched Browser Demo in chrome for andorid.

But I tried to connect my app's webview and I found it waiting defaultContext._loadDefaultContextAsIs(progress). By debugging, I found many errors:

{"error":{"code":-32601,"message":"'Page.enable' wasn't found"},"id":6}
{"error":{"code":-32601,"message":"'Page.getFrameTree' wasn't found"},"id":7}
{"error":{"code":-32601,"message":"'Log.enable' wasn't found"},"id":8}
{"error":{"code":-32601,"message":"'Page.setLifecycleEventsEnabled' wasn't found"},"id":9}
{"error":{"code":-32601,"message":"'Runtime.enable' wasn't found"},"id":10}
{"error":{"code":-32601,"message":"'Page.addScriptToEvaluateOnNewDocument' wasn't found"},"id":11}
...

and other information:

{ id: 1, method: 'Browser.getVersion', params: undefined }
{"id":1,"result":{"protocolVersion":"1.3","product":"Chrome/68.0.3440.91","revision":"@8daf58f7f40d22013c59388236c8e71e1117cb2c","userAgent":"Mozilla/5.0 (Linux; Android 8.0.0; MI 6 Build/OPR1.170623.027; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Mobile Safari/537.36","jsVersion":"6.8.275.26"}}

what are the possible reasons? Thanks

[pavelfeldman]

@kaivean: fixing the driver bundling, thanks for the heads up. As for the WebView, Chrome/68 is way too old, are you using ever-green WebView? WebView should be ever-green by default as of several years already on Android.

[pavelfeldman]

@javiercn: connectOverCDP works with WebView2. See https://gist.github.com/JoelEinbinder/e81d3533b9fcde7848251e35216487e8 as an example.

[kaivean]

@pavelfeldman Thanks for your reply. Yes, I used a old Chrome and fixed it. By the way, is there a plan support tests of ios emulators and real devices.