java.lang.OutOfMemoryError: Metaspace while running plugin tests on Travis CI

[martinbonnin]

Following https://github.com/gradle/gradle/issues/8354#issuecomment-529666227, a more detailed description of the problem:

This error happens systematically on Travis CI (works fine on my machine) when running the apollo-gradle-plugin-incubating tests:

./gradlew :apollo-gradle-plugin-incubating:test

After ~10min the tests fail (full logs at https://travis-ci.org/apollographql/apollo-android/builds/582181815):

com.apollographql.apollo.gradle.KotlinDSLTests > project configures correctly and generates something FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure: Unexpected build execution failure in /home/travis/build/apollographql/apollo-android/apollo-gradle-plugin-incubating/build/testProject with arguments [generateApolloClasses, --stacktrace]
    Output:
    Daemon will be stopped at the end of the build after running out of JVM memory
    FAILURE: Build failed with an exception.
    * What went wrong:
    A problem occurred configuring root project 'testProject'.
    > Could not open cache directory 5pj9vcbz7mfs0olb6tql927ze (/tmp/.gradle-test-kit-travis/caches/5.6/gradle-kotlin-dsl/5pj9vcbz7mfs0olb6tql927ze).
       > Metaspace

[...]

Caused by: java.lang.OutOfMemoryError: Metaspace
    	at org.jetbrains.kotlin.com.intellij.openapi.util.objectTree.ObjectNode.execute(ObjectNode.java:105)
    	at org.jetbrains.kotlin.com.intellij.openapi.util.objectTree.ObjectTree.executeAll(ObjectTree.java:162)
    	at org.jetbrains.kotlin.com.intellij.openapi.util.Disposer.dispose(Disposer.java:129)
    	at org.jetbrains.kotlin.com.intellij.openapi.util.Disposer.dispose(Disposer.java:125)
    	at org.gradle.kotlin.dsl.support.KotlinCompilerKt.compileToDirectory(KotlinCompiler.kt:624)
    	at org.gradle.kotlin.dsl.codegen.ApiExtensionsJarKt.compileKotlinApiExtensionsTo(ApiExtensionsJar.kt:113)
    	at org.gradle.kotlin.dsl.codegen.ApiExtensionsJarKt.compileKotlinApiExtensionsTo$default(ApiExtensionsJar.kt:130)
    	at org.gradle.kotlin.dsl.codegen.ApiExtensionsJarGenerator.compileExtensionsTo(ApiExtensionsJar.kt:64)
    	at org.gradle.kotlin.dsl.codegen.ApiExtensionsJarGenerator.generate(ApiExtensionsJar.kt:52)
    	at org.gradle.kotlin.dsl.codegen.ApiExtensionsJarKt.generateApiExtensionsJar(ApiExtensionsJar.kt:39)

The history of what I tried (Xmx4096m, -XX:MaxMetaspaceSize=512m, oraclejdk vs openjdk) is there: https://github.com/apollographql/apollo-android/pull/1577/commits

Most of the red crosses are MetaSpace OOMs except the 3 first ones which were a different issue. This commit for an example, should have the error: https://github.com/apollographql/apollo-android/pull/1577/commits/040770a5b3c686861a87d109975097c2029f49b7

Ultimately, I was able to make the tests pass by splitting them in 2 gradle invocations but that doesn't sound super robust: https://github.com/apollographql/apollo-android/pull/1577/commits/5ccdae8dbbff7cb64d556e8491b44741aaabcda8

The plugin tests use the GradleRunner APIs to run on 34 test projects. Most of them use groovy buildscripts but the one who fails uses build.gradle.kts kotlin buildscripts, that might be a lead.

[martinbonnin]

Build scan there: https://scans.gradle.com/s/vgoygq6oiacsy/

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

[martinbonnin]

Hi Stale Bot 👋 . Can we keep this one open a bit more? Finding the root of the MetaSpace error would allow to keep the daemon alive longer and have tests run faster

[big-andy-coates]

I'm seeing the OutOfMemoryError: Metaspace when running tests for a custom plugin on our build servers (AWS CodeBuild). Maybe the same thing.

[utsav0209]

@martinbonnin I am facing a similar issue with sonarqube. Have you found any workaround or root cause for this?

[martinbonnin]

@utsav0209 I ended up increasing the Java heap and metaspace:

https://github.com/apollographql/apollo-android/blob/59bc9ca38ea9bc4b9fbe2b01ba8f4d38340c260a/apollo-gradle-plugin/src/test/kotlin/com/apollographql/apollo3/gradle/util/TestUtils.kt#L31

[hql1991]

Saw this in a gradle project too. Upgrading from gradle 2.13 to 6.5 and such java metaspace error happen. Sometimes also complain heap exhausted.

Sus: saw Xmx Xms both set to 64m, but that shouldn't affect metaspace, it's in native memory...

[dsaff]

Just keeping this alive, saw this just today consistently on just the 8th invocation of GradleRunner, all fairly simple projects with build.gradle

[vlsi]

@martinbonnin , I see you attempted increasing metaspace via gradle.properties, however, it does not help me. Do you know if it really helped you?

What worked for me was casing GradleRunner to org.gradle.testkit.runner.internal.DefaultGradleRunner and adding metaspace:

this as DefaultGradleRunner
withJvmArguments("-XX:MaxMetaspaceSize=1g")

Just in case, here's a commit that passes org.gradle.jvmargs to teskit: https://github.com/sigstore/sigstore-java/pull/150

[martinbonnin]

Do you know if it really helped you?

I think it did ? I'm not 100% certain TBH but I haven't had any issue in a while now so I'm not really eager to open this pandora box... FWIW, the gradle.properties is in the project under test so I'm assuming the GradleRunner will be able to pick this one up too so it's technically equivalent to your commit. But since this not always reproducible, it's hard to tell for sure.

[vlsi]

the gradle.properties is in the project under test so I'm assuming the GradleRunner will be able to pick this one up too so it's technically equivalent to your commit

I tried creating gradle.properties in project under test, and it did not work for me for unknown reason.

[martinbonnin]

Thank for the update! I removed the -XX:MaxMetaspaceSize=1g from gradle.properties (here). I'll report here if the build fails again.