Can only fetch a single baseline profile from Firebase Test Lab

[scana]

Hi! 👋

I know that it's not directly related to running Macrobenchmarks but maybe someone has run into a similar issue with Firebase Test Lab.

My configuration is pretty basic, although I am relying on https://github.com/Flank/flank instead of direct gcloud calls so please do let me know if that's wrong place to open this issue.

environment-variables:
  androidx.benchmark.enabledRules: BaselineProfile
  no-isolated-storage: true
  additionalTestOutputDir: /sdcard/Download

I currently have two separate generators for baseline and startup profiles:

class BaselineProfileGenerator {

    @get:Rule
    val baselineProfileRule = BaselineProfileRule()

    @Test
    fun baselineProfile() {
        baselineProfileRule.collect(
            packageName = "me.scana.app",
            maxIterations = 5,
        ) {
            device.setCompressedLayoutHeirarchy(true)
            startActivityAndWait()
            // ...
            device.waitForIdle()
        }
    }
}

class StartupProfileGenerator {

    @get:Rule
    val baselineProfileRule = BaselineProfileRule()

    @Test
    fun startupProfile() {
        baselineProfileRule.collect(
            outputFilePrefix = "startup",
            packageName = "me.scana.app",
            includeInStartupProfile = true,
            maxIterations = 5,
        ) {
            device.setCompressedLayoutHeirarchy(true)
            startActivityAndWait()
            // ...
            device.waitForIdle()
        }
    }
}

How should I fetch them from Firebase Test Lab without having to run each test case separately?

Currently it seems like between each test run, /sdcard/Android/media/.../ gets cleared up in FTL, and ultimately only a single profile lands in a GCS bucket. Changing output to /sdcard/Download results in the same thing. Is this expected?

Logcat output suggests that both files are processed when tests run:

02-29 14:01:35.038: D/Benchmark(13352): Moving /storage/emulated/0/Android/media/me.scana.app/startup-startup-prof-2024-02-29-22-01-34.txt to /sdcard/Download/startup-startup-prof-2024-02-29-22-01-34.txt

02-29 14:02:39.667: D/Benchmark(14079): Moving /storage/emulated/0/Android/media/me.scana.app/BaselineProfileGenerator_baselineProfile-baseline-prof-2024-02-29-22-02-39.txt to /sdcard/Download/BaselineProfileGenerator_baselineProfile-baseline-prof-2024-02-29-22-02-39.txt

If I run them separately, each one will properly download a profile file. If I launch both test cases separately within Android Studio it seems that at least default directory (media/...) gets cleaned up each time a test finishes.

[keyboardsurfer]

You can check this configuration and see how that works. I'm seeing multiple profiles dropped into the gcloud bucket: https://github.com/android/performance-samples/blob/main/.github/workflows/firebase_test_lab.yml