`@Tag` at the class level results in test number mismatch

[compscidr]

If I use the @Tag annotation on a per-method basis, filtering works great, however, if I add the annotation at the class level, it seems to report a difference between the expected tests, and the received tests:

Starting 2 tests on Pixel 4a (5G) - 13
Test run failed to complete. Expected 2 tests, received 1

> Task :app:connectedDebugAndroidTest FAILED

FAILURE: Build failed with an exception.

For instance these tests work: https://github.com/compscidr/hello-kotlin-android/blob/main/app/src/androidTest/java/com/example/myapplication/ExampleInstrumentedTest.kt

but if I add a new class and a difference test tag at the class level, running the same tests will fail. I've got my filters setup like this: https://github.com/compscidr/hello-kotlin-android/blob/main/app/build.gradle#L65

        junitPlatform {
            String includedTagsProperty = System.getProperty("includeTags")
            boolean includedTagsPropertySet = includedTagsProperty != null

            String excludedTagsProperty = System.getProperty("excludeTags")
            boolean excludedTagsPropertySet = excludedTagsProperty != null

            if (includedTagsPropertySet && excludedTagsPropertySet) {
                filters("debug") {
                    includeTags includedTagsProperty
                    excludeTags excludedTagsProperty
                }
            } else if (includedTagsPropertySet && !excludedTagsPropertySet) {
                filters("debug") {
                    includeTags includedTagsProperty
                }
            } else if (!includedTagsPropertySet && excludedTagsPropertySet) {
                filters("debug") {
                    excludeTags excludedTagsProperty
                }
            }
        }

So I can do something like ./gradlew connectedAndroidTest -DincludeTags="TAGTEST" to filter to one of the test tags. However as I add a new class like this:

package com.example.myapplication

import org.junit.jupiter.api.Test
import org.slf4j.LoggerFactory

// @Tag("Test") // this won't work
class SomeOtherTaggedTest {
    private val logger = LoggerFactory.getLogger(javaClass)

    @Test fun test() {
        logger.debug("SomeOtherTaggedTest")
    }
}

The same test which had previously been passing, now fails.

Happy to contribute to a fix if there's any suggestions on where to look at.

[compscidr]

Update: It actually appears that class level @Tag does work, if you only have a single test class in all of your tests. When you introduce the second class, whether there are class level @Tag annotations or not, you get the mismatch failure.

[compscidr]

Also, looks like if I try to use both includeTags and excludeTags together, I'm getting some type of parsing error:

junitPlatform {
            filters {
                includeTags 'TEST1'
                excludeTags 'TEST2'
            }
        }

com.example.myapplication.ExampleInstrumentedTest > initializationError[Pixel 4a (5G) - 13] FAILED 
        org.junit.platform.commons.PreconditionViolationException: Unable to parse tag expression "TEST1-T TEST2": missing operator between 'TEST1-T' at index <6> and 'TEST2' at index <8>
        at org.junit.platform.launcher.TagFilter.lambda$parse$8(TagFilter.java:181)

Perhaps this is a separate issue though.

[mannodermaus]

Hey, thanks for reporting and apologies for the late reply. It does seem like two distinct issues that you have identified here, but they might be tangentially related. I'll try to take a look!

[compscidr]

Thanks, let me know if I can help at all, happy to contribute if I can.

[mannodermaus]

A quick update and a note for myself: I managed to find a potential breakthrough for this ticket while dealing with the linked #316 issue. It should be possible to avoid prematurely reporting a test class to the Android instrumentation that would end up being skipped when JUnit 5 gets to it later. Stay tuned for more! 👀

[mannodermaus]

This has been merged via https://github.com/mannodermaus/android-junit5/pull/326 and will be available in the upcoming version 1.5.0 of the instrumentation test libraries. If you need the fix right now, add the Sonatype Snapshots repository to your project and grab the 1.5.0-SNAPSHOT instead 🙇‍♂️