What's the recommended configuration to get Testify working on Firebase Cloud Test Lab?

[DanielJette]

What's the recommended configuration to get Testify working on Firebase Cloud Test Lab?

[DanielJette]

Copied from https://github.com/Shopify/android-testify/issues/247#issuecomment-952150343

cc @udeyrishi

I'm currently trying this locally:

system_image="system-images;android-29;google_apis;x86_64"
device="pixel_4"

and sending this flag to the firebase gloud command on CI: --device=model=flame,version=29,locale=en_US

That's yielding slightly different images:

Local	Firebase

cc @Angelia-Gong

[DanielJette]

Copied from https://github.com/Shopify/android-testify/issues/247#issuecomment-952152525

In order to run the screenshot tests successfully, both locally and on Firebase CI, it is important that you maintain an emulator specifically designed for screenshot testing. The Testify Screenshot library requires that you use an identically configured Emulator AVD locally and on CI.

AVD setup

1. In Android Studio, select Tools :arrow_right: AVD Manager :arrow_right: Create Virtual Device ...
2. Select Pixel 2 (1080x1920 420dpi)
3. Select Q, API level 29, x86, Android 10.0 (Google APIs)
4. Under the hardware configuration, verify:
   • RAM: 1536 MB
   • VM heap: 256 MB
   • Internal Storage: 2048 MB
   • SD card, Studio-managed: 512 MB
   • Enable Device Frame with pixel_2 skin
   • Enable keyboard input

⚠️ Note that Android Studio creates the Pixel 2 with the incorrect DPI. To fix this, launch the emulator and run adb shell wm density 441 from your terminal.

Runtime emulator configuration

Once the emulator is booted:

1. Run adb shell wm density 441 from your terminal
2. Set the Language to English (United States) (en_US)
3. In the developer settings, set Window animation scale, Transition animation scale, and Animator duration scale to Off

You only need to run these commands once.

Firebase

On Firebase, you want to use the following configuration:

• Pixel2,29,en_US,portrait
• "form": "VIRTUAL"

Set the following environment variables on Firebase:

• TESTIFY_USE_SDCARD=true
• useSdCard=true

Failed screenshots will be posted to:

/storage/emulated/0/Android/data/<<your.package.name>>/files/testify_images/

[DanielJette]

Another tip:

Run the following command locally on your emulator: ./gradlew app:testifyKey

Then, look in the Firebase logs for the tag Baseline could not be found in <<key>>.

Check that the testifyKey matches what's being output in the logs.

[DanielJette]

@Angelia-Gong let me know if this helps/doesn't help. I've found this configuration I posted here works best with Firebase and is a good mid-range device with a reasonable screen resolution. Plus, it's one of the virtual (cheaper) devices on Firebase.

[DanielJette]

Copied from https://github.com/Shopify/android-testify/issues/247#issuecomment-952335755

@DanielJette Hi Dan, I have followed the configurations above and the images still look slightly different (notice the colour difference of the top bar. One is greyer and one is darker). At least that's the only difference to my eyes

local	firebase

[DanielJette]

@Angelia-Gong 🤔 I haven't seen this particular issue before.

My guess is that it looks like some different resources are loading locally compared to on Firebase.

Do you have any alternative resource qualifiers on your project? like -hdpi or -night. Is it possible that Firebase is running with a dark/night theme enabled? (Or perhaps running with dark theme locally?)

Is this a new project based on the recent Android Studio template projects? The most recent templates auto-create a themes.xml in the values-night directory. So, maybe try removing the night theme?

[DanielJette]

Copied from https://github.com/Shopify/android-testify/issues/247#issuecomment-952368465

@DanielJette No we don't have night mode. It's not a very recent project and there's no themes.xml in values-night. Would it be related to the virtual GPU Google uses?

[DanielJette]

@DanielJette No we don't have night mode. It's not a very recent project and there's no themes.xml in values-night. Would it be related to the virtual GPU Google uses?

@Angelia-Gong I've not encountered this problem myself, but I could certainly see this as possible. If there's alpha blending going on with the colour being used by the status bar/toolbar then this could happen. Are you using a non-opaque colour for either of those elements?

One suggestion you may want to consider is to see if you can get the Sample app from this repository running on your CI environment. Whether that's successful or not could indicate whether there's an issue in the Testify library, Firebase or the app code itself.

Also, if you wanted to share the code for the test and the Hello world activity you're trying, I could run it on my environment and see if I get the same results.