How to run the plugin on Firebase Test Lab? (or external CI tool like Bitrise)

[ddudek]

Hi guys,

I'm trying to run verifying screenshots on firebase test lab, is it possible? can anyone share steps that I need to do to achieve that? It's not clear from readme.

[sanginovs]

@ddudek I think these are the steps:

1. Store initial screenshot somewhere (e.g S3 bucket)
2. Run screenshot tests in Firebase test lab and copy all the output into CircleCI.
3. CircleCI should compare initial screenshot with Firebase test lab output to make sure they match.

@xiphirx any suggestion on this?

[tdrhq]

(Edited in 2022)

Here's a full-fledged example of running screenshot-tests-for-android with Firebase test lab: https://github.com/screenshotbot/screenshot-tests-for-android-example

But it's based off a tool we built, https://screenshotbot.io. Modern documentation here: https://screenshotbot.io/documentation/platforms/android-apps

[sanginovs]

Hi @tdrhq quick question. Can screenshotbot run tests on Firebase test lab devices that have API version > 23?

[tdrhq]

@sanginovs Screenshotbot doesn't actually run the tests, it just collects the screenshots and creates a nice workflow after the fact (i.e. catching regressions etc.) I have tested against API level 27 (If you look at https://github.com/tdrhq/screenshotbot-example/blob/master/.circleci/config.yml, and look at line 33, you'll see me specifying API level 27). API Level 28 and 29 are a bit more tricky currently with Firebase because of https://github.com/facebook/screenshot-tests-for-android/issues/260 (looks like you created that task, is there a specific reason you mentioned API version 23?)

[sanginovs]

@tdrhq my tests were failing when i was running them on api level 23+ but i tested again a few days ago and it works now. I had another question. Currently, there's an issue when you generate screenshots locally on emulator on a certain device model + API level and then compare them against Firebase generated screenshots, they fail even if they are running on the same emulator device model + API level. Did you run into this issue? Here's the error i get:

The size of the screenshot taken has changed. Original screenshot dimension: 1080x1794. New recorded screenshot dimension: 1080x1788.

[tdrhq]

@sanginovs That's weird, I haven't seen that yet. Do you use custom measuring/layout logic for that view or is it all stock views? One easy thing to check, go to your Test Lab in Firebase and look at your previous runs and make sure that all your devices are in portrait mode. (AFAIK, it should be in portrait unless you explicitly specify landscape, but there maybe something about your test runs that I'm not aware of. In theory portrait/landscape shouldn't affect your view snapshots but it could)

[sanginovs]

@tdrhq i was able to resolve this issue by specifying extra info about Pixel device in CI: Before, i just specified model as Pixel2 and os version 27 in CI. I added extra info about pixel device like this and it worked: model=walleye,version=27,locale=en_US,orientation=portrait

i think test lab has different versions of pixel2 device

[tdrhq]

@sanginovs Very cool, good to know. That's a virtual device and not a real device, correct?

[sanginovs]

yes virtual. i think pixel2 physical device is deprecated.