Improve ActivityScenario support so we can take the screenshots from the onActivity method

[pedrovgs]

Time ago, Espresso core deprecated ActivityTestRule in favor or the ActivityScenario feature which is way more flexible. We should provide proper support for this new API ensuring at least one of our composers use this API. Documentation should also be updated.

[drewhamilton]

Right now running in scenario.onActivity causes a crash: "java.lang.RuntimeException: java.lang.RuntimeException: This method can not be called from the main application thread"

This makes some sense, because the onActivity callback runs on the main thread. Through the stacktrace it's clear where it goes wrong: The default impl of runOnUi calls Intrumentation.runOnMainSync, which fails if you're already on the main thread. To work around this crash, I can override runOnUI:

override fun runOnUi(block: () -> Unit) {
    if (Thread.currentThread() == Looper.getMainLooper().thread)
        block()
    else
        super.runOnUi(block)
}

Then another error appears. Same message, this time pointing to waitForAnimationsToFinish, called by disableFlakyComponentsAndWaitForIdle. Again, I can override this and skip the "waitForIdle" bits from the default implementation (I also have to skip some private method calls but hopefully those are unimportant 😬):

override fun disableFlakyComponentsAndWaitForIdle(view: View) {
    if (Thread.currentThread() == Looper.getMainLooper().thread)
        prepareUIForScreenshot()
    else
        super.disableFlakyComponentsAndWaitForIdle(view)
}

Now the test runs to completion!

But record mode is broken.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':demo:debugDownloadScreenshots'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:205)
	at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:203)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:184)
	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: java.lang.RuntimeException: Nonzero exit value: 1
	at scala.sys.package$.error(package.scala:27)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:134)
	at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:105)
	at com.karumi.shot.android.Adb.executeAdbCommandWithResult(Adb.scala:45)
	at com.karumi.shot.android.Adb.pullScreenshots(Adb.scala:35)
	at com.karumi.shot.Shot.$anonfun$pullScreenshots$1(Shot.scala:147)
	at com.karumi.shot.Shot.$anonfun$pullScreenshots$1$adapted(Shot.scala:142)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at com.karumi.shot.Shot.pullScreenshots(Shot.scala:142)
	at com.karumi.shot.Shot.downloadScreenshots(Shot.scala:37)
	at com.karumi.shot.tasks.DownloadScreenshotsTask.downloadScreenshots(Tasks.scala:101)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:42)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:727)
	at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:694)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:568)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:553)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:536)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$300(ExecuteActionsTaskExecuter.java:109)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:276)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:265)
	at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$1(ExecuteStep.java:33)
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)
	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:67)
	at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:36)
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:49)
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:34)
	at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:43)
	at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)
	at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)
	at org.gradle.internal.execution.steps.CatchExceptionStep.execute(CatchExceptionStep.java:34)
	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:44)
	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:54)
	at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:38)
	at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:49)
	at org.gradle.internal.execution.steps.CacheStep.executeWithoutCache(CacheStep.java:159)
	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:72)
	at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:43)
	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:44)
	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:33)
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:38)
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:24)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:92)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:85)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:39)
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:76)
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:37)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:36)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:26)
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:94)
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:49)
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:79)
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:53)
	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:74)
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:78)
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:78)
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:39)
	at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:40)
	at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:28)
	at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:192)
	... 30 more

This failure may be related to one or more of a few things:

• I didn't call those private methods when overriding disableFlakyComponentsAndWaitForIdle.
• The test I'm running is the only one in the suite.

Any ideas for what to look at next?

If it helps, here's the full source of my test:

class DemoTest : ScreenshotTest {

    @MediumTest
    @Test fun snapshotTest() {
        val scenario = ActivityScenario.launch(DemoAppActivity::class.java)
        scenario.onActivity { activity ->
            compareScreenshot(activity)
        }
    }

    override fun runOnUi(block: () -> Unit) = if (isOnMainThread) block() else super.runOnUi(block)

    override fun disableFlakyComponentsAndWaitForIdle(view: View) = if (isOnMainThread)
        prepareUIForScreenshot()
    else
        super.disableFlakyComponentsAndWaitForIdle(view)

    private val isOnMainThread: Boolean
        get() = Thread.currentThread() == Looper.getMainLooper().thread
}

[pedrovgs]

This is really helpful @drewhamilton !! Thank you so much 😃 This issue is on my tasks' list and I'll try to fix it soon. However, I have to finish some tasks before 😃

[pedrovgs]

I've just found an ugly but working workaround:

    // Hack needed until we fully support Activity Scenarios
    private fun waitForActivtiyScenario(): MainActivity {
        var activity: MainActivity? = null
        activityScenarioRule.scenario.onActivity {
            activity = it
        }
        while (activity == null) {
            Log.d("MainActivityTest", "Waiting for activity to be initialized")
        }
        return activity!!
    }

So you can write tests like this:

class MainActivityTest : ScreenshotTest {

    @get:Rule
    var activityScenarioRule = activityScenarioRule<MainActivity>()

    @Test
    fun activityTest() {
        val activity = waitForActivtiyScenario()

        compareScreenshot(activity)
    }

    // Hack needed until we fully support Activity Scenarios
    private fun waitForActivtiyScenario(): MainActivity {
        var activity: MainActivity? = null
        activityScenarioRule.scenario.onActivity {
            activity = it
        }
        while (activity == null) {
            Log.d("MainActivityTest", "Waiting for activity to be initialized")
        }
        return activity!!
    }
}

[drewhamilton]

True! This is what we're doing in the meantime. But the ActivityScenario documentation explicitly says not to do this so we'd better add real support before the ActivityScenario police find this thread.

Incidentally, you don't need the while loop because onActivity blocks while waiting for the UI thread operation to finish (internally using Instrumentation.waitForComplete).

[pedrovgs]

Thanks for the tip @drewhamilton However, keep in mind I'm the only core contributor working on this project and I have no time to develop all the features the users would expect to be implemented in this library during office hours. I'm afraid right now I'm working on Jetpack Compose support and I have no time to provide ActivityScenario support until I finish this task. Feel free to contribute if you are interested in this feature, any contribution is more than welcome.

[drewhamilton]

Yep totally get it. I'll try a contribution soon, just worried I'll get stuck since I don't know the code base. The workaround suffices in the meantime.

[pedrovgs]

@drewhamilton as I finished Jetpack Compose support on time I had time today to implement this feature. I've done my best :( https://github.com/Karumi/Shot/pull/156

[drewhamilton]

Just my 2¢: I would recommend against releasing ActivityScenarioUtils.waitForActivity as part of your public API. It seems way outside of Shot's responsibilities to get an Activity instance from an ActivityScenario instance—this has nothing to do with screenshots. And it's already possible for ActivityScenario consumers to do this. Plus, as discussed above, it's a workaround against ActivityScenario documentation recommendations.

I also wouldn't close this issue, since it's still not possible to compareScreenshot inside the launch lambda, which is what consumers would expect. TestNameDetector is a problem, but not the primary problem with running on the main thread. I could live with the wrong class name since I can pass my own name suffix anyway. The real problem—and the place I got stuck trying to contribute—is that the "record" functionality fails on the main thread—particularly the debugDownloadScreenshots task as shown in my original stacktrace.

[pedrovgs]

I'm afraid we need to add this extension method to the library @drewhamilton . There is no way we can get the library working without the workaround. Even if you configure the screenshot name and I've fixed the threading issues, deep down inside the library the test class name and the test name are used to build the screenshot's metadata. Thank you so much for your feedback and help anyway.

[drewhamilton]

Hmm, is that these lines in Facebook's library? Or is there more to it?

If so I'd like to file an issue and/or PR to that repo to make TestNameDetector pluggable in some way. And then Shot could document that ActivityScenario consumers must provide a custom instance of it that doesn't rely on the current thread (or it could provide such an implementation by default). What do you think of this idea?

[pedrovgs]

These lines and also the new jetpack compose support. Your solution is ideal. Making TestNameDetector pluggable, but we should keep the public API as simple as possible in the future. I'm going to leave this issue open as you mentioned to see if we can improve this scenario 😃