java.lang.OutOfMemoryError: Java heap space

[timrijckaert]

Describe the bug Hello,

As we have been adding screenshot tests to more and more Android modules we wanted to parallelise our CI build to first build the test APK's. After all modules have created their APK we run ./gradlew runFlank

We have about 14 modules thus-far containing screenshot tests.
Averaging in size of approximately 60MB

The build stops with a OutOfMemory.
See full logs in Build summary section below.

Build summary

Task :configureFulladle Task :validateFladleConfig Task :writeConfigProps Task :execFlank version: v21.11.0 revision: 1bf9b6a8ed1287141b66a75ec66b23813a02f2a8 session id: 08044f2b-9127-402c-9a6a-f5b6e7b51c13 AndroidArgs gcloud: results-bucket: test-lab-nxzauzcrb50h6-m6yq6i7d7dwni results-dir: 2022-07-11_06-29-04.516092_hCGC record-video: false timeout: 15m async: false client-details: network-profile: null results-history-name: null # Android gcloud app: /xxx/xxxx/xxxxx/testlab/testlab-debug.apk test: /xxx/xxxx/xxxxx/features/xxxx/build/outputs/apk/androidTest/debug/xxxx-debug-androidTest.apk additional-apks: auto-google-login: false use-orchestrator: false directories-to-pull: - /storage/emulated/0/Download/screenshots/ grant-permissions: all type: null other-files: scenario-numbers: scenario-labels: obb-files: obb-names: performance-metrics: false num-uniform-shards: null test-runner-class: null test-targets: - annotation com.xxxx.xxxxx.xxxxx.ShotTest robo-directives: robo-script: null device: - model: Pixel2 version: 29 locale: en orientation: portrait num-flaky-test-attempts: 0 test-targets-for-shard: fail-fast: false parameterized-tests: default flank: max-test-shards: 1 shard-time: -1 num-test-runs: 1 smart-flank-gcs-path: smart-flank-disable-upload: false default-test-time: 120.0 use-average-test-time-for-new-tests: false files-to-download: - .*.png$ test-targets-always-run: disable-sharding: false project: neba-android local-result-dir: /xxx/xxxx/xxxxx/flank full-junit-result: false # Android Flank Yml keep-file-path: false additional-app-test-apks: - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/features/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk - test: /xxx/xxxx/xxxxx/libraries/xxxxx/build/outputs/apk/androidTest/debug/xxxxx-debug-androidTest.apk max-test-shards: 5 run-timeout: 30m legacy-junit-result: false ignore-failed-tests: false output-style: verbose disable-results-upload: false default-class-test-time: 240.0 disable-usage-statistics: false output-report: none skip-config-validation: false custom-sharding-json: RunTests Smart Flank cache hit: 0% (0 / 23) Shard times: 2760s Smart Flank cache hit: 0% (0 / 1) Shard times: 120s Smart Flank cache hit: 0% (0 / 37) Shard times: 4440s Smart Flank cache hit: 0% (0 / 8) Shard times: 960s Smart Flank cache hit: 0% (0 / 3) Shard times: 360s Smart Flank cache hit: 0% (0 / 9) Shard times: 1080s Smart Flank cache hit: 0% (0 / 2) Shard times: 240s Smart Flank cache hit: 0% (0 / 4) Shard times: 480s Smart Flank cache hit: 0% (0 / 3) Shard times: 360s Smart Flank cache hit: 0% (0 / 1) Shard times: 120s Smart Flank cache hit: 0% (0 / 13) Shard times: 1560s Smart Flank cache hit: 0% (0 / 5) Shard times: 600s Smart Flank cache hit: 0% (0 / 211) Shard times: 5040s, 5040s, 5040s, 5040s, 5160s Saved 13 shards to /xxx/xxxx/xxxxx/flank/android_shards.json Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/... Uploading [testlab-debug.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.. Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/.. Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/...... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/........................... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/... Uploading [xxxxx-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-nxzauzcrb50h6-m6yq6i7d7dwni/2022-07-11_06-29-04.516092_hCGC/..... java.lang.OutOfMemoryError: Java heap space at java.base/java.nio.file.Files.read(Files.java:3153) at java.base/java.nio.file.Files.readAllBytes(Files.java:3213) at ftl.util.FileReferenceKt.upload(FileReference.kt:33) at ftl.util.FileReferenceKt.uploadIfNeeded(FileReference.kt:28) at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:35) at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:27) at ftl.run.platform.android.UploadApksKt.access$upload(UploadApks.kt:1) at ftl.run.platform.android.UploadApksKt$upload$2$1$1.invokeSuspend(UploadApks.kt:23) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) Total run duration: 2m 40s - Preparation: 0m 6s Task :execFlank FAILED FAILURE: Build failed with an exception.

• What went wrong: Execution failed for task ':execFlank'.

Process 'command '/usr/lib/jvm/java-11-openjdk-amd64/bin/java'' finished with non-zero exit value 15

• Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org BUILD FAILED in 6m 27s Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings 4 actionable tasks: 4 executed

java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.file.Files.read(Files.java:3153)
	at java.base/java.nio.file.Files.readAllBytes(Files.java:3213)
	at ftl.util.FileReferenceKt.upload(FileReference.kt:33)
	at ftl.util.FileReferenceKt.uploadIfNeeded(FileReference.kt:28)
	at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:35)
	at ftl.run.platform.android.UploadApksKt.upload(UploadApks.kt:27)
	at ftl.run.platform.android.UploadApksKt.access$upload(UploadApks.kt:1)
	at ftl.run.platform.android.UploadApksKt$upload$2$1$1.invokeSuspend(UploadApks.kt:23)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Any idea on how to fix this?

Additional context

• Running on Gitlab shared runners
• Flank version: v21.11.0
• I saw possible related ticket

Thanks in advance.

[bootstraponline]

@timrijckaert Hey, for this use case I'd recommend uploading the apks to Google Cloud Storage via gsutil, and then passing Flank the GCS path to the apks. That'll ensure there's no memory problems as the apks get larger over time. The tests will also be a bit faster as the apk only has to be uploaded once.

Alternatively, you could try increasing the Java heap space made available to flank. Something like: java -jar -Xms2048M -Xmx2048M flank.jar

[timrijckaert]

Hi @bootstraponline

Thanks for the quick reply! I'm using the Fladle plugin. Any idea or more detailed information on how to configure it to use a combination of gsutil and the plugin?

Or pass the memory options to the runFlank task?

Thanks in advance

[bootstraponline]

@timrijckaert I don't think Fladle supports the gsutil strategy I mentioned.

Try to increase the heap space by setting gradle.properties https://stackoverflow.com/a/47454640

[timrijckaert]

Hmm same result unfortunately

[bootstraponline]

@timrijckaert I'd open an issue on https://github.com/runningcode/fladle/issues to see if they can advise.

[bootstraponline]

@timrijckaert The fladle maintainer suggested to set the maxHeapSize on the FlankExec task:

https://docs.gradle.org/current/dsl/org.gradle.api.tasks.JavaExec.html#org.gradle.api.tasks.JavaExec:maxHeapSize

[timrijckaert]

I applied the following

task flankMaxHeapSize(type: com.osacky.flank.gradle.FlankJavaExec) {
    maxHeapSize = "4g"
}

I get

Task :flankMaxHeapSize FAILED Error: Could not find or load main class ftl.Main Caused by: java.lang.ClassNotFoundException: ftl.Main

[loki666]

issue seems to be related to the Files.readAllBytes() call when a file need to be uploaded https://github.com/Flank/flank/blob/3748bce448d27af9cfdd9f9f534e5e5a6a53a34d/test_runner/src/main/kotlin/ftl/util/FileReference.kt#L33

maybe a new type should be introduced (ie: RemoteStorage.File instead of RemoteStorage.Data which takes the ByteArray) to delay the loading and use a buffered read/write with GcStorage (ie: using a WriterChannel)

[timrijckaert]

To add on this

https://github.com/Flank/flank/blob/33d6f7ad5efca7daff70e59a7c0177c06bc0207d/test_runner/src/main/kotlin/ftl/run/platform/android/UploadApks.kt#L56-L62

This gets amplified by the amount of APK's to upload.

[bootstraponline]

@loki666 @timrijckaert If you'd like to submit a patch, I'll happily review it.