Set up Android Github Action pipeline

[reta]

Coming from https://github.com/mockito/mockito/issues/2890#issuecomment-1409200309:

we started an effort to set up some basic Android tests for us: https://github.com/mockito/mockito/tree/main/subprojects/androidTest However, this stalled as we couldn't get it to work on CI. If anybody affected here would like to dedicate some time to prevent these kind of issues from happening, please take a look at the existing test suite and infrastructure and submit a PR to get it passing on CI.

The presence of such Github Action would have helped to prevent regressions like https://github.com/mockito/mockito/issues/2890.

@TimvdLippe I will try to pull it off (I don't work with Android sadly), it seems like Github Actions have support for it.

[TimvdLippe]

I am seeing some failures on CI with the new API 33 level job:

> Task :androidTest:connectedDebugAndroidTest
Exception thrown during onBeforeAll invocation of plugin com.google.testing.platform.plugin.android.AndroidDevicePlugin.
Failed to install APK(s): /Users/runner/work/mockito/mockito/subprojects/androidTest/build/outputs/apk/androidTest/debug/androidTest-debug-androidTest.apk
Unknown failure: cmd: Can't find service: package
com.android.ddmlib.InstallException: Unknown failure: cmd: Can't find service: package
	at com.android.ddmlib.internal.DeviceImpl.installRemotePackage(DeviceImpl.java:1315)
	at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1141)
	at com.android.tools.utp.plugins.deviceprovider.ddmlib.DdmlibAndroidDevice.installPackage(DdmlibAndroidDevice.kt)
	at com.android.tools.utp.plugins.deviceprovider.ddmlib.DdmlibAndroidDeviceController$executeAsync$deferred$1.invokeSuspend(DdmlibAndroidDeviceController.kt:173)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

From https://github.com/mockito/mockito/actions/runs/4106509220/jobs/7084834115

@TWiStErRob do you know what is going on here?

[TWiStErRob]

I have an idea, will raise a PR a bit later. You can rerun, and it should very likely work the second time.

[TWiStErRob]

Created #2903, hopefully it will help. I've got many builds as you can see on the commit history, none of them failed for any weird reason, so I'm hopeful.

[TimvdLippe]

Unfortunately we got another failure, this time on 26: https://github.com/mockito/mockito/actions/runs/4150981851/jobs/7180984789

[TWiStErRob]

🙄 I'm starting to think it's related to the time of day...

[reta]

🙄 I'm starting to think it's related to the time of day...

Should we try to reduce just to one Android version for now? It won't solve the issue but we could have less often CI failures while we are looking for solution(s) to make it stable?

[TWiStErRob]

I would say let's leave it for a week or two, and see actually how bad it is. For now we see it failed once... out of 2 executions. That could be 1% or 50%, depending on how many you measure with.

If it's too bad, I guess we could move to be part of the release process to reduce the burden, but keep the safety the tests provide.

Or we could add retry the same way you have retry for normal tests, which was surprising to see.

[TimvdLippe]

Unfortunately another failure on main: https://github.com/mockito/mockito/actions/runs/4168588270/jobs/7215565389#step:4:706

Failed to install APK(s): /Users/runner/work/mockito/mockito/subprojects/androidTest/build/outputs/apk/androidTest/debug/androidTest-debug-androidTest.apk
Unknown failure: cmd: Can't find service: package
com.android.ddmlib.InstallException: Unknown failure: cmd: Can't find service: package

[reta]

@TWiStErRob @TimvdLippe so what should we do? I would still try to reduce Android matrix to one version, to minimize the number of possible build failures ...

[TWiStErRob]

I would try to add retry first, but up to you. We could reduce it to 1 API version, which would still fail sometimes, or move the whole thing to release time, and run it once at the end of the cycle; or somewhere in the middle, running it scheduled in the background on main, but then who/when will look at the results?

@chrisbanes as the main tamer of Android UI libs how do you think we can reduce flakiness on this trivial Android test project? The problem is that the emulator doesn't start up quite often, the actual tests are fine so far.

CI: https://github.com/mockito/mockito/blob/main/.github/workflows/ci.yml#L89

Project: https://github.com/mockito/mockito/tree/main/subprojects/androidTest

Example failures: #2903's OP ("Investigation" section, first 3)

[chrisbanes]

To be honest, I've had very little issues with GitHub Actions (other than starting the emulator is slow).

I'd recommend sticking to the default images, and google_apis if necessary.

You might also want to explore Gradle Managed Devices. https://developer.android.com/studio/test/gradle-managed-devices

It simplifies the setup greatly, but you need to use recent AGP versions for recent fixes.

[TimvdLippe]

The Android 33 job fails the most (2 instances tonight, with the latest being https://github.com/mockito/mockito/actions/runs/4231340587/jobs/7361785524). Therefore, I am inclined to remove Android 33 for now, as Android 26 appears stable for now.

[TWiStErRob]

I found a way to consistently trigger a quick boot on the emulator from a clean state (WIP), this might solve issues 1-3 listed in https://github.com/mockito/mockito/pull/2903 and potentially could help with 4 as well, since the emulator always starts from a consistent state. The tradeoff: ~2GB of cache for the emulator Disk and RAM.

Example cache:

Example quick boot:

Compared to a "slow boot"

which might or might not time out.

Note: the slow boot still needs to be performed probably once a week to fill the cache, but the chance of failure should go way down.

I'll raise a PR to change the CI once I'm done with my workflow change and used it a bit "in production".

[reta]

This is great, thanks a lot for the update @TWiStErRob !