android
ActivityScenario (unlike ActivityTestRule) does not mirror the real application usage
For the project I'm working on with my colleagues we've used ActivityTestRule for a long time with our Espresso tests. Then we've started using Compose and because of that we had to use ComposeTestRule more and more often. Recently I've noticed problems with some of our tests and once I've investigated it, I've noticed something really weird. The activity (don't know about fragments, but I've focused on activities) lifecycle is not the same during the tests run with ComposeTestRule as it is when I run app myself and mirror the steps from the test. To simplify, I have a Splash, Home and Settings activities. The test is doing this: Runs the app from Splash, then the app navigates from splash to home (with clearing the stack which explains why SplashActivity is destroyed before Home), then it navigates to Settings where one setting is forcing the app to recreate itself, then goes back to Home and exits the app by back button.
I've printed logs in onCreate, onResume (for Home) and onDestroy. This is what I get when I run test using ActivityTestRule: SplashActivity#onCreate SplashActivity#onDestroy HomeActivity#onCreate HomeActivity#onResume SettingsActivity#onCreate (here the app recreation is initiated) HomeActivity#onDestroy HomeActivity#onCreate HomeActivity#onResume SettingsActivity#onCreate SettingsActivity#onDestroy (here we exit Settings) HomeActivity#onResume SettingsActivity#onDestroy (here we exit the app) HomeActivity#onDestroy
This is exactly what one would expect following those steps and that is exactly what I get when I do that myself after running the app. But when now we run this test using ComposeTestRule, this test is flaky and fails from time to time. On top of it, even if it succeeds, the lifecycle is not what one would expect.
This is what is printed when the test fails: SplashActivity#onCreate HomeActivity#onCreate HomeActivity#onResume SettingsActivity#onCreate SplashActivity#onDestroy SplashActivity#onCreate HomeActivity#onDestroy HomeActivity#onCreate HomeActivity#onResume HomeActivity#onDestroy HomeActivity#onCreate HomeActivity#onResume SettingsActivity#onDestroy HomeActivity#onDestroy SplashActivity#onDestroy
This is what happens when the test succeeds: SplashActivity#onCreate HomeActivity#onCreate HomeActivity#onResume SettingsActivity#onCreate SplashActivity#onDestroy SplashActivity#onCreate HomeActivity#onDestroy HomeActivity#onCreate HomeActivity#onResume HomeActivity#onDestroy SettingsActivity#onCreate SettingsActivity#onDestroy HomeActivity#onCreate HomeActivity#onResume SettingsActivity#onDestroy SplashActivity#onDestroy HomeActivity#onDestroy
The main question is, why when tests being run with ActivityScenario are showing different lifecycle of the app comparing to normal usage? When I've noticed the above and I've stopped using createAndroidComposeRule() but instead created AndroidComposeTestRule myself using ActivityTestRule instead of ActivityScenarioRule, the lifecycle is the same as when using the app manually and the test passes 100% of the time and never is flaky.
What's going on?
Steps to Reproduce
Explained above
Expected Results
The app lifecycle when running the tests using ActivityScenario should be exactly the same as when running the app myself and doing the same steps.
Actual Results
There are differences in lifecycle.
AndroidX Test and Android OS Versions
androidx.test.ext:junit and androidx.test.ext:junit-ktx 1.1.2 but I've also checked the recent 1.1.4-alpha07. Android Api 29, 30 and 32.
