StackOverflow when derivation of large case class fails only when using scalac directly

[WojciechMazur]

Based on OpenCB failure in com-lihaoyi/upickle - build logs

FYI @lihaoyi

I've started to notice this problems recently. Compilation using mill sometimes passes, sometimes it does not.

For some reason when compiling snippet below we can use the default stack size -Xss1M which is enough to compile the code when using compiler server under the hood. However, if we'll try to compile it directly (scala compile --server=false) if fails to compile even if we increate the stack size 100x.

To consider:

• are -Xss / JAVA_TOOL_OPTIONS options correctly passed inside the scala-cli?
• is forking for macros execution using the correct JVM options (are -Xss etc, inherited the forked program)

Similar problem can be found in: dacr/sotohp - build logs

Compiler version

Scala 3.3 (all versions) Scala 3.8.0-RC3 Scala 3.8.1 nightly

Reproduced code

//> using dep com.lihaoyi::upickle:4.4.1

given upickle.default.ReadWriter[Big150] = upickle.default.macroRW

case class Big150(__0: Int = 31337, __1: Int, __2: Int, __3: Int, __4: Int, __5: Int, __6: Int, __7: Int,
                  __8: Int, __9: Int, __10: Int, __11: Int, __12: Int, __13: Int, __14: Int,
                  __15: Int, __16: Int, __17: Int, __18: Int, __19: Int, __20: Int, __21: Int,
                  __22: Int, __23: Int, __24: Int, __25: Int, __26: Int, __27: Int, __28: Int,
                  __29: Int, __30: Int, __31: Int, __32: Int, __33: Int, __34: Int, __35: Int,
                  __36: Int, __37: Int, __38: Int, __39: Int, __40: Int, __41: Int, __42: Int,
                  __43: Int, __44: Int, __45: Int, __46: Int, __47: Int, __48: Int, __49: Int,
                  __50: Int, __51: Int, __52: Int, __53: Int, __54: Int, __55: Int, __56: Int,
                  __57: Int, __58: Int, __59: Int, __60: Int, __61: Int, __62: Int, __63: Int,
                  __64: Int, __65: Int, __66: Int, __67: Int, __68: Int, __69: Int, __70: Int,
                  __71: Int, __72: Int, __73: Int, __74: Int, __75: Int, __76: Int, __77: Int,
                  __78: Int, __79: Int, __80: Int, __81: Int, __82: Int, __83: Int, __84: Int,
                  __85: Int, __86: Int, __87: Int, __88: Int, __89: Int, __90: Int, __91: Int,
                  __92: Int, __93: Int, __94: Int, __95: Int, __96: Int, __97: Int, __98: Int,
                  __99: Int, __100: Int, __101: Int, __102: Int, __103: Int, __104: Int,
                  __105: Int, __106: Int, __107: Int, __108: Int, __109: Int, __110: Int,
                  __111: Int, __112: Int, __113: Int, __114: Int, __115: Int, __116: Int,
                  __117: Int, __118: Int, __119: Int, __120: Int, __121: Int, __122: Int,
                  __123: Int, __124: Int, __125: Int, __126: Int, __127: Int, __128: Int,
                  __129: Int, __130: Int, __131: Int, __132: Int, __133: Int, __134: Int,
                  __135: Int, __136: Int, __137: Int, __138: Int, __139: Int, __140: Int,
                  __141: Int, __142: Int, __143: Int, __144: Int, __145: Int, __146: Int,
                  __147: Int, __148: Int, __149: Int = -1337 /*default*/){
}

Output

Using compile server:

> scala bloop exit
Stopped Bloop server.

> time JAVA_TOOL_OPTIONS=-Xss1M scala compile test.scala -S 3.nightly
Picked up JAVA_TOOL_OPTIONS: -Xss1M
Compiling project (Scala 3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY, JVM (21))
Compiled project (Scala 3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY, JVM (21))
JAVA_TOOL_OPTIONS=-Xss1M scala compile test.scala -S 3.nightly  3.65s user 0.36s system 53% cpu 7.558 total

Using no compile servier:

> JAVA_TOOL_OPTIONS=-Xss100M scala compile test.scala -S 3.nightly --server=false
Picked up JAVA_TOOL_OPTIONS: -Xss100M
Picked up JAVA_TOOL_OPTIONS: -Xss100M

  unhandled exception while running staging on /Users/wmazur/projects/scala/community-build3/test.scala

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.


     while compiling: /Users/wmazur/projects/scala/community-build3/test.scala
        during phase: staging
                mode: Mode(ImplicitsEnabled)
     library version: version 3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY
    compiler version: version 3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY-git-c6051b4
            settings: -classpath /Users/wmazur/Library/Caches/Coursier/v1/https/repo.scala-lang.org/artifactory/maven-nightlies/org/scala-lang/scala3-library_3/3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY/scala3-library_3-3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/upickle_3/4.4.1/upickle_3-4.4.1.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo.scala-lang.org/artifactory/maven-nightlies/org/scala-lang/scala-library/3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY/scala-library-3.8.1-RC1-bin-20251206-c6051b4-NIGHTLY.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/ujson_3/4.4.1/ujson_3-4.4.1.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/upack_3/4.4.1/upack_3-4.4.1.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/upickle-implicits_3/4.4.1/upickle-implicits_3-4.4.1.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/upickle-core_3/4.4.1/upickle-core_3-4.4.1.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/geny_3/1.1.1/geny_3-1.1.1.jar -d /Users/wmazur/projects/scala/community-build3/.scala-build/community-build3_9cd75c3c84-9d23cdd7be/classes/main -sourceroot /Users/wmazur/projects/scala/community-build3

Exception in thread "main" java.lang.StackOverflowError
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1542)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
        at dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1589)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
        at dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1549)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
        at dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1555)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
        at dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1589)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)
        at dotty.tools.dotc.staging.CrossStageSafety.transform(CrossStageSafety.scala:123)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1589)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:67)
        at dotty.tools.dotc.staging.TreeMapWithStages.transform(TreeMapWithStages.scala:46)

Expectation