Parameterized Android tests on Robolectric could not be run

[tiwiz]

Description

I had the need to test a specific feature of a paging component that used Room, and I needed to use a real in memory version of Room, so I needed the Context. Since the tests cases needed to cover different inputs, but the logic was the same, I intended to use Parameterized tests from JUnit 4

Steps to Reproduce

Open the SimpleParameterizedTest.kt class and try to execute it with Robolectric.

Actual Results

If we run the suite with @RunWith(AndroidJUnit4::class), we get the following error:

java.lang.RuntimeException: Delegate runner 'org.robolectric.RobolectricTestRunner' for AndroidJUnit4 could not be loaded.

	at androidx.test.ext.junit.runners.AndroidJUnit4.throwInitializationError(AndroidJUnit4.java:92)
	at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:82)
	at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:51)
	at androidx.test.ext.junit.runners.AndroidJUnit4.<init>(AndroidJUnit4.java:46)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
	at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
	at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
	at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:49)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:72)
	... 16 more
Caused by: org.junit.runners.model.InitializationError
	at org.junit.runners.ParentRunner.validate(ParentRunner.java:418)
	at org.junit.runners.ParentRunner.<init>(ParentRunner.java:84)
	at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:65)
	at org.robolectric.internal.SandboxTestRunner.<init>(SandboxTestRunner.java:56)
	at org.robolectric.RobolectricTestRunner.<init>(RobolectricTestRunner.java:92)
	... 21 more

If we run the suite with @RunWith(Parameterized::class), we instead get the following error:

java.lang.IllegalStateException: No instrumentation registered! Must run under a registering instrumentation.

	at androidx.test.platform.app.InstrumentationRegistry.getInstrumentation(InstrumentationRegistry.java:45)
	at androidx.test.core.app.ApplicationProvider.getApplicationContext(ApplicationProvider.java:41)
	at net.orgiu.tests.SimpleParameterizedTest.<init>(SimpleParameterizedTest.kt:15)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.junit.runners.parameterized.BlockJUnit4ClassRunnerWithParameters.createTestUsingConstructorInjection(BlockJUnit4ClassRunnerWithParameters.java:43)
	at org.junit.runners.parameterized.BlockJUnit4ClassRunnerWithParameters.createTest(BlockJUnit4ClassRunnerWithParameters.java:38)
	at org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:266)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

AndroidX Test and Android OS Versions

I am trying with the following dependencies

testImplementation 'androidx.test:runner:1.1.1'
testImplementation 'androidx.test.espresso:espresso-core:3.1.1'
testImplementation 'androidx.test.ext:junit:1.1.0'
testImplementation 'org.robolectric:robolectric:4.1'

Link to a public git repo demonstrating the problem:

GitHub Repository

[jongerrish]

Hey @tiwiz So, this is kind of tough right now...

• Robolectric needs its own test runner in order to create its own sandbox environment to execute Android code, so to write parameterized tests in Robolectric you need to use @RunWith(RobolectricParameterizedTestRunner.class)
• Instrumentation tests don't have that restriction so you can legitimately use @RunWith(Parameterized.class)
• The unified test runner is AndroidJUnit4.class which will detect the environment in which you run and delegate to either Robolectric or Instrumentation under the covers. However, it just supports regular JUnit4 style tests, not parameterized.

Long term Nitrogen will solve this as we won't need any custom JUnit4 test runners to bootstrap Robolectric or Instrumentation tests (which will be done via N2 itself.

So... what can we do in the meantime?

I chatted with the team and we think creating a ParameterizedAndroidJUnit4.class which will would replace RobolectricParameterizedTestRunner.class and also delegate to the regular Parameterized.class to work on-device too.

This helps with our goal of consolidating Robolectric test runners into AndroidX Test and it will also remain compatible (although not necessary) once Nitrogen + Sheep device are shipped.

What do you think about having a go at implementing this ParameterizedAndroidJUnit4 test runner? We're not going to get to it right very soon, but we'd provide full support in guidance and code reviews and getting it into a release. You'd get all the glory :-)

[tiwiz]

Thanks for clarifying! I'll give it a shot :-)

[mrk-han]

@jongerrish Appreciate the thoughtful response here. Is there any update on the release of N2? Anxiously waiting to get my hands on it. Would love to help contribute if any of it is open source.

[jongerrish]

@mrk-han Hey Mark, the team is still working hard on N2 and we're trialing it internally, mainly the effort to externalize it has been dependent on the integration into studio which is the next phase we're working on presently, because its such a fundamental change its something we really need to make sure we get right.

[jongerrish]

@tiwiz awesome - feel free to ping me if you need anything for this!

[jongerrish]

Hi @tiwiz were you able to make any progress on this issue?

[tiwiz]

Unfortunately I was not 😶 overwhelmed with work stuff, but I plan to do it ASAP, hopefully before the summer

[mrctrifork]

And here we are one year later 😢

[mrctrifork]

Here's how you do it as per 4/06/2020

@RunWith(ParameterizedRobolectricTestRunner::class)
@Config(sdk = [Build.VERSION_CODES.O_MR1] )
class CypherUnitTest(
    val param1: Double,
    val param2: Double,
    val param3: Double,
    val param4: String,
    val param5 String
) {

      companion object {
             private fun mold(
                  val param1: Double,
                  val param2: Double,
                  val param3: Double,
                  val param4: String,
                  val param5 String
            ) = arrayOf(param1, param2, param3, param4, param5)
           
            @JvmStatic
            @ParameterizedRobolectricTestRunner.Parameters
             fun testCases() = listOf(
                   mold(…),
                   mold(…),
                   mold(…)
             )
      }

     @Test
      fun yourTest(){ … }

}

[jameswald]

That worked for me. Looks like this issue could be closed?

[utzcoz]

That worked for me. Looks like this issue could be closed?

If those tests only run Robolectric, we can use ParameterizedRobolectricTestRunner by following https://github.com/android/android-test/issues/283#issuecomment-638705886. But I found @tiwiz 's example uses sharedTest pattern to share tests between Robolectric and instrumentation tests. The ParameterizedAndroidJUnit4, that can run correctly on Robolectric and instrumentation tests, maybe needed for this occasion.

[alixwar]

The ParameterizedAndroidJUnit4, that can run correctly on Robolectric and instrumentation tests, maybe needed for this occasion.

Here is an ugly hack to get it to work (while we wait for something better...). I provide the recipe instead of the code as I'm not sure about the license implications:

1. Copy-paste androidx.test.ext.junit.runners.AndroidJUnit4.java (from artifact androidx.test.ext:junit:androidx-test-junit) to a new file: ParameterizedAndroidJUnit4.java
2. In ParameterizedAndroidJUnit4.java: Replace string occurrences of org.robolectric.RobolectricTestRunner with org.robolectric.ParameterizedRobolectricTestRunner
3. Annotate your parameterized test class with: @RunWith(ParameterizedAndroidJUnit4::class)

Done.

[sergio-sastre]

@jongerrish @utzcoz Now that Robolectric supports screenshot testing, and screenshot tests are more likely to need parameters, what about giving this a higher priority to allow the same screenshot test run either on a device/emulator or on the JVM?

I could give it a shot once back from holidays, in April. I believe the challenging part is to make it work for instrumented tests, @alixwar workaround should, with some adjustments, suffice for getting it working with robolectric

[utzcoz]

@sergio-sastre @alixwar Maybe you can check https://android-review.googlesource.com/c/platform/platform_testing/+/2623172, the CL for ParameterizedAndroidJUnit4. @brettchabot Will you have a plan to public it or does it exist in the AXT repository already?

[sergio-sastre]

@sergio-sastre @alixwar Maybe you can check https://android-review.googlesource.com/c/platform/platform_testing/+/2623172, the CL for ParameterizedAndroidJUnit4. @brettchabot Will you have a plan to public it or does it exist in the AXT repository already?

Actually I had something simpler in mind that's what I am using in this ParameterizedRobolectricSharedTestRunner for my library and it is working:

Since the main issue is that the current ParameterizedRobolectricTestRunner requires its own Parameters annotation i.e. @ParameterizedRobolectricTestRunner.Parameters

1. copy-pasted the ParameterizedRobolectricTestRunner
2. replace @ParameterizedRobolectricTestRunner.Parameters uses with @ParameterizedTestRunner.Parameters

I understand that's not enough to avoid breaking changes but one could write the ParameterizedRobolectricTestRunner to support both annotations.

But... just my 2 cents