StackOverflowError from KAPT on building Arrow

[abelkov]

trafficstars

When I try to build the current master of Arrow (e610ca6677f9bc912691144437abb8e5fbb1b50b) I get a StackOverflowError from kapt. I'm on Ubuntu 18.04. My coworker on macOS doesn't have this problem. This is currently not blocking me (I don't need to build Arrow anymore), just filing the issue for reference. CC @nomisRev

Here is the truncated build output:

➜  arrow git:(master) java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

➜  arrow git:(master) ./gradlew clean build
Configuration(s) specified but the install task does not exist in project :arrow-benchmarks-fx.
Configuration(s) specified but the install task does not exist in project :arrow-kio-benchmarks.
Configuration(s) specified but the install task does not exist in project :arrow-meta-test-models.
Configuration(s) specified but the install task does not exist in project :arrow-scala-benchmarks.

> Task :arrow-meta:kaptKotlin
w: [kapt] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: com.google.auto.service.processor.AutoServiceProcessor (NON_INCREMENTAL).

> Task :arrow-kindedj:kaptKotlin
w: [kapt] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: arrow.core.extensions.ExtensionProcessor (NON_INCREMENTAL), arrow.fold.AutoFoldProcessor (NON_INCREMENTAL), arrow.generic.CoproductProcessor (NON_INCREMENTAL), arrow.generic.ProductProcessor (NON_INCREMENTAL), arrow.higherkinds.HigherKindsProcessor (NON_INCREMENTAL), arrow.optics.OpticsProcessor (NON_INCREMENTAL).

> Task :arrow-core-data:kaptKotlin
w: [kapt] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: arrow.core.extensions.ExtensionProcessor (NON_INCREMENTAL), arrow.fold.AutoFoldProcessor (NON_INCREMENTAL), arrow.generic.CoproductProcessor (NON_INCREMENTAL), arrow.generic.ProductProcessor (NON_INCREMENTAL), arrow.higherkinds.HigherKindsProcessor (NON_INCREMENTAL), arrow.optics.OpticsProcessor (NON_INCREMENTAL).

> Task :arrow-core-data:compileKotlin

[some warnings]

> Task :arrow-core:kaptKotlin FAILED
e: [kapt] An exception occurred: java.lang.StackOverflowError
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:447)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at java.lang.StringBuilder.append(StringBuilder.java:76)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:484)
        at java.lang.StringBuilder.append(StringBuilder.java:166)
        at java.lang.StringBuilder.append(StringBuilder.java:76)
        at kotlin.sequences.SequencesKt___SequencesKt.joinTo(_Sequences.kt:1719)
        at kotlin.sequences.SequencesKt___SequencesKt.joinToString(_Sequences.kt:1743)
        at kotlin.sequences.SequencesKt___SequencesKt.joinToString$default(_Sequences.kt:1742)
        at kotlin.text.StringsKt__StringsJVMKt.replace(StringsJVM.kt:76)
        at kotlin.text.StringsKt__StringsJVMKt.replace$default(StringsJVM.kt:75)
        at arrow.meta.encoder.jvm.StringTypeExtensionsKt.removeVariance(StringTypeExtensions.kt:9)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.toMeta(KotlinPoetEncoder.kt:71)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.typeNameToMetaImpl(KotlinPoetEncoder.kt:154)
        at arrow.meta.encoder.jvm.TypeElementEncoder$DefaultImpls.typeNameToMetaImpl(TypeElementEncoder.kt)
        at arrow.meta.encoder.jvm.JvmMetaApi$DefaultImpls.typeNameToMetaImpl(JvmMetaApi.kt)
        at arrow.meta.processor.MetaProcessor.typeNameToMetaImpl(MetaProcessor.kt:33)
        at arrow.meta.processor.MetaProcessor$typeNameToMeta$1.invoke(MetaProcessor.kt:50)
        at arrow.meta.processor.MetaProcessor$typeNameToMeta$1.invoke(MetaProcessor.kt:33)
        at arrow.meta.internal.MemoizeKey1.invoke(memoization.kt:15)
        at arrow.meta.internal.MemoizeKey1.invoke(memoization.kt:14)
        at arrow.meta.internal.MemoizedHandler.invoke(memoization.kt:20)
        at arrow.meta.internal.MemoizationKt$memoize$1.invoke(memoization.kt:7)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.toMeta(KotlinPoetEncoder.kt:147)
        at arrow.meta.encoder.jvm.TypeElementEncoder$DefaultImpls.toMeta(TypeElementEncoder.kt)
        at arrow.meta.encoder.jvm.JvmMetaApi$DefaultImpls.toMeta(JvmMetaApi.kt)
        at arrow.meta.processor.MetaProcessor.toMeta(MetaProcessor.kt:33)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.toMeta(KotlinPoetEncoder.kt:94)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.typeNameToMetaImpl(KotlinPoetEncoder.kt:152)
        at arrow.meta.encoder.jvm.TypeElementEncoder$DefaultImpls.typeNameToMetaImpl(TypeElementEncoder.kt)
        at arrow.meta.encoder.jvm.JvmMetaApi$DefaultImpls.typeNameToMetaImpl(JvmMetaApi.kt)
        at arrow.meta.processor.MetaProcessor.typeNameToMetaImpl(MetaProcessor.kt:33)
        at arrow.meta.processor.MetaProcessor$typeNameToMeta$1.invoke(MetaProcessor.kt:50)
        at arrow.meta.processor.MetaProcessor$typeNameToMeta$1.invoke(MetaProcessor.kt:33)
        at arrow.meta.internal.MemoizeKey1.invoke(memoization.kt:15)
        at arrow.meta.internal.MemoizeKey1.invoke(memoization.kt:14)
        at arrow.meta.internal.MemoizedHandler.invoke(memoization.kt:20)
        at arrow.meta.internal.MemoizationKt$memoize$1.invoke(memoization.kt:7)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.toMeta(KotlinPoetEncoder.kt:147)
        at arrow.meta.encoder.jvm.TypeElementEncoder$DefaultImpls.toMeta(TypeElementEncoder.kt)
        at arrow.meta.encoder.jvm.JvmMetaApi$DefaultImpls.toMeta(JvmMetaApi.kt)
        at arrow.meta.processor.MetaProcessor.toMeta(MetaProcessor.kt:33)
        at arrow.meta.encoder.jvm.KotlinPoetEncoder$DefaultImpls.toMeta(KotlinPoetEncoder.kt:136)

        [etc...]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':arrow-core:kaptKotlin'.
> Compilation error. See log for more details

Here is the full build output with --info logging: https://gist.github.com/abelkov/98a6a0e23e097b076a4e0a90c7c38aa9

[rachelcarmena]

Thanks @abelkov! However, I think we could look at something else. I'm on Ubuntu 18.04 and I don't have that problem. We're using Ubuntu 18.04 on GitHub environments and it also works properly:

Run cat /proc/version
Linux version 5.0.0-1027-azure (buildd@lgw01-amd64-033) 
(gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) 
arrow-kt/arrow#29~18.04.1-Ubuntu SMP Mon Nov 25 21:18:57 UTC 2019

Run java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (Zulu 8.42.0.23-CA-linux64) (build 1.8.0_232-b18)
OpenJDK 64-Bit Server VM (Zulu 8.42.0.23-CA-linux64) (build 25.232-b18, mixed mode)

Please, does it happen with previous commits? Thanks in advance!

[rachelcarmena]

And please, check your running processes, Gradle daemons, ...

[abelkov]

Yes, it also fails on some previous commits and on a different JDK 8 from Azul, and when I run gradle with --no-daemon. Given that it's not blocking me and nobody else seems to have this problem, I think we can ignore this for now. Thanks.

[rachelcarmena]

Yes, it also fails on some previous commits and on a different JDK 8 from Azul, and when I run gradle with --no-daemon. Given that it's not blocking me and nobody else seems to have this problem, I think we can ignore this for now. Thanks.

Thank @abelkov so much for the info!! There is another person with the same problem. Let's see if we find some clue...

[calvellido]

It also happens to me (similar environment), but I haven't been able to dig deeper on what could be the source for this :cry:

[calvellido]

Current tests are also failing due to a StackOverflow error, though I don't know how related these two are:

https://github.com/arrow-kt/arrow/pull/1934/checks?check_run_id=394918808#step:4:4960

[1Jajen1]

@calvellido that looks like something different: @nomisRev should take a look at that ^^

[nomisRev]

@1Jajen1 thanks for the poke! @calvellido that is definitely something else, looks like a bad merge to me. I will take a look asap ;)

[rachelcarmena]

I was reviewing this issue with @nomisRev because he got an error with JitPack.io.

We observed that the same commit produced different results:

• https://jitpack.io/com/github/arrow-kt/arrow/0.10.4/build.log
• https://jitpack.io/com/github/arrow-kt/arrow/d5d623338d1ed16271bbbb7f5422ce7a053ed96b/build.log

The only difference is the assigned machine.

After doing more checks with other commits, we observed that JitPack.io assigned machines in a random way:

• Linux 4.10.0-28-generic amd64
• Linux 4.14.63-xxxx-std-ipv6-64 amd64
• Linux 4.18.0-13-generic amd64

Just it works with 4.10.0-28-generic amd64.

More data from personal dev environments:

• KO: Linux 4.15.0-74-generic amd64
• OK: Linux 5.3.0-26-generic amd64

And GitHub Actions environment for ubuntu-latest: Linux 5.0.0-1028-azure amd64

[calvellido]

I tried:

• 4.15.0-74 :x:
• 4.15.0-72 :x:
• 4.15.0-70 :x:
• 4.10.0-38 :x:
• 4.8.0-53 :x:

All of them leading to the same error:

> Task :arrow-core:kaptKotlin FAILED
e: [kapt] An exception occurred: java.lang.StackOverflowError

:sob:

[rachelcarmena]

Hi! this could be tried again with the new organization. Thanks!!

[calvellido]

Still failing for me after the split :sob:

• 4.15.0-88 :x:

[aballano]

After seeing https://github.com/arrow-kt/arrow-core/issues/91 it seems to me that there might be some recursive calling in the Meta processors that some systems are able to handle due to bigger heap memory than others, that's why it could be happening "randomly" on CI based on https://github.com/arrow-kt/arrow-core/issues/25#issuecomment-576779353 cc @raulraja @i-walker

[raulraja]

After seeing https://github.com/arrow-kt/arrow-core/issues/91 it seems to me that there might be some recursive calling in the Meta processors that some systems are able to handle due to bigger heap memory than others, that's why it could be happening "randomly" on CI based on https://github.com/arrow-kt/arrow-core/issues/25#issuecomment-576779353 cc @raulraja @i-walker

That is the case, recursion in our kapt processors is not stack safe. It will also not terminate in some cases if you make complex recursive type nesting. We don't plan to fix it as we have reencoded most of them in meta or will be superceded soon by this new features.