Extracting UI tests leak reports back to Gradle's Test task output

[gmazzo]

Problem description

In the Leak detection in UI tests section, states some snippets on how use this tool at scale, mostly integrated with Bugsnag.

Since we are starting to explore the tool ourselves in our CI, it would be nice to have a way to export the generated Leak reports from the device and later use them to report back to the CI itself (in our case GitHub Actions).

Potential solutions

By leveraging JUnit's org.junit.runner.notification.RunListener and that they are automatically added by AndroidJUnitRunner if a META-INF/services entry is present in the classpath, I came up with a zero-config approach for extracting the generated report in (.txt and .json formats, plus the .hprof heap dump for further analysis) out from the device as Android's Gradle Test task output:

This enables us our use case, by just collecting any file matching the pattern, and reporting back to the PR.

I was wondering if you would be open to a contribution on this feature, it will probably require a dedicated Gradle plugin to wire all together:

• Add behavior customization through DSL:
  • when to run LeakCanary: after each test, each suite, all tests
  • make the tests fail or pass if leaks are found
  • options to turn on/off each kind of report: .hprof, .txt, .json (using opinionated Gson library)
• Add dedicated Gradle tasks to collect the output from AGP's Test tasks, and let them available at a given location
• Further manipulation into other report formats like: checkstyle or sarif

Additional information

The content of the CanaryLeakJUnitReporter.kt file:

import android.util.Log
import androidx.test.platform.app.InstrumentationRegistry
import com.google.gson.ExclusionStrategy
import com.google.gson.FieldAttributes
import com.google.gson.GsonBuilder
import leakcanary.AndroidDetectLeaksAssert
import org.junit.runner.Description
import org.junit.runner.notification.Failure
import org.junit.runner.notification.RunListener
import shark.HeapAnalysisSuccess
import java.util.concurrent.ConcurrentSkipListSet
import kotlin.io.path.Path
import kotlin.io.path.copyTo
import kotlin.io.path.createDirectories
import kotlin.io.path.writeText
import kotlin.io.path.writer

@RunListener.ThreadSafe
class CanaryLeakJUnitReporter : RunListener() {

    private val failed = ConcurrentSkipListSet<Description>()

    override fun testFailure(failure: Failure) {
        failed.add(failure.description)
    }

    override fun testFinished(description: Description) {
        super.testFinished(description)

        if (!failed.remove(description)) {
            AndroidDetectLeaksAssert(heapAnalysisReporter = {
                if (it is HeapAnalysisSuccess && it.applicationLeaks.isNotEmpty()) {
                    reportLeak(description, it)
                }
            }).assertNoLeaks(description.displayName)
        }
    }

    private fun reportLeak(description: Description, analysis: HeapAnalysisSuccess) {
        val baseName = "${description.className}.${description.methodName}"
        val toExportAnalysis = analysis.copy(
            heapDumpFile = additionalTestOutputDir.resolve("$baseName.${analysis.heapDumpFile.extension}").toFile()
        )

        // copies the .hprof file
        analysis.heapDumpFile.toPath()
            .copyTo(toExportAnalysis.heapDumpFile.toPath())

        // generates a .txt file with the leak trace
        additionalTestOutputDir.resolve("$baseName.txt")
            .writeText(toExportAnalysis.toString())

        // generates a .json file with the leak trace
        additionalTestOutputDir.resolve("$baseName.json")
            .writer().use { gson.toJson(toExportAnalysis, it) }

        Log.i("LeakCanary", "Leak report generated for $baseName")
    }

    private companion object {

        private val additionalTestOutputDir by lazy {
            Path(checkNotNull(InstrumentationRegistry.getArguments().getString("additionalTestOutputDir")) {
                "Missing expected instrumentation argument 'additionalTestOutputDir'"
            }).resolve("leakcanary").apply { createDirectories() }
        }

        private val gson by lazy {
            GsonBuilder()
                .setPrettyPrinting()
                .addSerializationExclusionStrategy(object : ExclusionStrategy {
                    override fun shouldSkipClass(clazz: Class<*>) = false
                    override fun shouldSkipField(field: FieldAttributes) =
                        field.name == "heapDumpFile"
                })
                .create()
        }

    }

}