graal
graal copied to clipboard
Published
20 hours ago •
oracle
oracle
Runtime exception when running Dotty's native image
Describe the issue When compiling Dotty using native-image then running the resulting executable, I get a runtime exception. Tried with @mukel.
Steps to reproduce the issue
Setup and compilation:
➜ ~ export JAVA_HOME=/Library/Java/JavaVirtualMachines/graalvm-jdk-21.0.2+13.1/Contents/Home
➜ ~ export PATH=/Library/Java/JavaVirtualMachines/graalvm-jdk-21.0.2+13.1/Contents/Home/bin:$PATH
➜ ~ which java
/Library/Java/JavaVirtualMachines/graalvm-jdk-21.0.2+13.1/Contents/Home/bin/java
➜ ~ java -version
java version "21.0.2" 2024-01-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30, mixed mode, sharing)
➜ ~ native-image --version
native-image 21.0.2 2024-01-16
GraalVM Runtime Environment Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30)
Substrate VM Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS, serial gc, compressed references)
➜ ~ git clone https://github.com/lampepfl/dotty dotty-fresh
➜ ~ cd dotty-fresh
➜ ~/dotty-fresh git:(main) sbt "dist / pack"
...
[info] welcome to sbt 1.9.7 (Oracle Corporation Java 21.0.2)
...
➜ ~/dotty-fresh git:(main) sbt "export scala3-library / fullClasspath"
...
/Users/mbovel/dotty-fresh/library/target/scala-3.3.1/classes:/Users/mbovel/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar
➜ ~/dotty-fresh git:(main) export SCALA_LIB="/Users/mbovel/dotty-fresh/library/target/scala-3.3.1/classes:/Users/mbovel/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar"
Check that we can actually run Dotty with java:
➜ ~/dotty-fresh git:(main) echo '@main def main = println("Hello world")' > hello.scala
➜ ~/dotty-fresh git:(main) java -cp 'dist/target/pack/lib/*' dotty.tools.dotc.Main -cp $SCALA_LIB hello.scala
➜ ~/dotty-fresh git:(main) java -cp $SCALA_LIB:. main
Hello world
Run Dotty with the tracing agent to generate native-image-config:
➜ ~/dotty-fresh git:(main) java -cp 'dist/target/pack/lib/*' -agentlib:native-image-agent=config-output-dir=native-image-config dotty.tools.dotc.Main -cp $SCALA_LIB hello.scala
➜ ~/dotty-fresh git:(main) ls native-image-config
agent-extracted-predefined-classes proxy-config.json serialization-config.json
jni-config.json reflect-config.json
predefined-classes-config.json resource-config.json
Compile Dotty using native-image:
➜ ~/dotty-fresh git:(main) native-image -cp 'dist/target/pack/lib/*' -H:+UnlockExperimentalVMOptions -H:ConfigurationFileDirectories=native-image-config dotty.tools.dotc.Main
==========================================================================================================
GraalVM Native Image: Generating 'dotty.tools.dotc.main' (executable)...
==========================================================================================================
[1/8] Initializing... (13.0s @ 0.14GB)
Java version: 21.0.2+13-LTS, vendor version: Oracle GraalVM 21.0.2+13.1
Graal compiler: optimization level: 2, target machine: x86-64-v3, PGO: off
C compiler: cc (apple, x86_64, 15.0.0)
Garbage collector: Serial GC (max heap size: 80% of RAM)
2 user-specific feature(s):
- com.oracle.svm.polyglot.scala.ScalaFeature
- com.oracle.svm.thirdparty.gson.GsonFeature
----------------------------------------------------------------------------------------------------------
Build resources:
- 12.09GB of memory (75.6% of 16.00GB system memory, determined at start)
- 12 thread(s) (100.0% of 12 available processor(s), determined at start)
[2/8] Performing analysis... [****] (51.7s @ 1.60GB)
17,675 reachable types (91.1% of 19,393 total)
28,777 reachable fields (75.1% of 38,304 total)
92,912 reachable methods (55.7% of 166,914 total)
3,047 types, 311 fields, and 2,717 methods registered for reflection
59 types, 58 fields, and 56 methods registered for JNI access
4 native libraries: -framework Foundation, dl, pthread, z
[3/8] Building universe... (8.3s @ 2.05GB)
[4/8] Parsing methods... [***] (6.9s @ 2.00GB)
[5/8] Inlining methods... [***] (2.8s @ 2.30GB)
[6/8] Compiling methods... [*********] (91.3s @ 2.44GB)
[7/8] Layouting methods... [***] (9.8s @ 3.64GB)
[8/8] Creating image... [***] (8.7s @ 4.64GB)
42.42MB (57.25%) for code area: 49,810 compilation units
31.38MB (42.35%) for image heap: 325,743 objects and 121 resources
308.82kB ( 0.41%) for other data
74.10MB in total
----------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
26.19MB s.4.1-RC1-bin-SNAPSHOT.jar 13.42MB byte[] for code metadata
8.02MB java.base 5.47MB byte[] for java.lang.String
3.51MB scala-library-2.13.12.jar 3.55MB java.lang.Class
2.85MB svm.jar (Native Image) 2.35MB java.lang.String
508.10kB scala-asm-9.6.0-scala-1.jar 972.60kB byte[] for reflection metadata
245.95kB jdk.zipfs 828.52kB com.oracle.svm.core.hub.DynamicHubCompanion
179.87kB jdk.crypto.ec 652.23kB byte[] for general heap data
147.15kB com.oracle.svm.svm_enterprise 534.78kB c.o.s.c.hub.DynamicHub$ReflectionMetadata
138.94kB java.logging 500.47kB heap alignment
83.19kB jdk.charsets 443.42kB byte[] for embedded resources
229.35kB for 15 more packages 2.74MB for 3080 more object types
Use '-H:+BuildReport' to create a report with more details.
----------------------------------------------------------------------------------------------------------
Security report:
- Binary includes Java deserialization.
- Use '--enable-sbom' to embed a Software Bill of Materials (SBOM) in the binary.
----------------------------------------------------------------------------------------------------------
Recommendations:
PGO: Use Profile-Guided Optimizations ('--pgo') for improved throughput.
INIT: Adopt '--strict-image-heap' to prepare for the next GraalVM release.
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
QBM: Use the quick build mode ('-Ob') to speed up builds during development.
----------------------------------------------------------------------------------------------------------
16.1s (8.3% of total time) in 173 GCs | Peak RSS: 5.92GB | CPU load: 8.71
----------------------------------------------------------------------------------------------------------
Produced artifacts:
/Users/mbovel/dotty-fresh/dotty.tools.dotc.main (executable)
==========================================================================================================
Finished generating 'dotty.tools.dotc.main' in 3m 13s.
Trying to run Dotty's native image to compile hello.scala:
➜ ~/dotty-fresh git:(main) ./dotty.tools.dotc.main -cp $SCALA_LIB hello.scala
Exception while compiling hello.scala
An unhandled exception was thrown in the compiler.
Please file a crash report here:
https://github.com/lampepfl/dotty/issues/new/choose
For non-enriched exceptions, compile with -Yno-enrich-error-messages.
while compiling: <no file>
during phase: <some phase>
mode: Mode()
library version: version 2.13.12
compiler version: version 3.4.1-RC1-bin-SNAPSHOT-git-f92349b
settings: -classpath /Users/mbovel/dotty-fresh/library/target/scala-3.3.1/classes:/Users/mbovel/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar
Exception in thread "main" java.lang.AssertionError: assertion failed: asTerm called on not-a-Term val <none>
at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
at dotty.tools.dotc.core.Symbols$Symbol.asTerm(Symbols.scala:171)
at dotty.tools.dotc.core.Definitions.ObjectClass(Definitions.scala:324)
at dotty.tools.dotc.core.Definitions.ObjectType(Definitions.scala:328)
at dotty.tools.dotc.core.Definitions.AnyRefAlias(Definitions.scala:427)
at dotty.tools.dotc.core.Definitions.syntheticScalaClasses(Definitions.scala:2134)
at dotty.tools.dotc.core.Definitions.syntheticCoreClasses(Definitions.scala:2147)
at dotty.tools.dotc.core.Definitions.init(Definitions.scala:2163)
at dotty.tools.dotc.core.Contexts$ContextBase.initialize(Contexts.scala:900)
at dotty.tools.dotc.core.Contexts$Context.initialize(Contexts.scala:523)
at dotty.tools.dotc.Run.rootContext(Run.scala:464)
at dotty.tools.dotc.Run.<init>(Run.scala:485)
at dotty.tools.dotc.Compiler.newRun(Compiler.scala:172)
at dotty.tools.dotc.Driver.doCompile(Driver.scala:35)
at dotty.tools.dotc.Driver.process(Driver.scala:196)
at dotty.tools.dotc.Driver.process(Driver.scala:164)
at dotty.tools.dotc.Driver.process(Driver.scala:176)
at dotty.tools.dotc.Driver.main(Driver.scala:206)
at dotty.tools.dotc.Main.main(Main.scala)
Printing the help page works however:
➜ ~/dotty-fresh git:(main) ./dotty.tools.dotc.main --help
Usage: scalac <options> <source files>
where possible standard options include:
...
For the record I also tried to compile it through Scala CLI's --native-image options, yielding the same result:
➜ ~/dotty-fresh git:(main) scala-cli --power package -o dotty-native --native-image --main-class dotty.tools.dotc.Main --classpath 'dist/target/pack/lib/*' --graalvm-args '-cp dist/target/pack/lib/*' --graalvm-args "--no-fallback"
By the way, not directly related to the issue, but there actually is a much nicer possible setup using Scala CLI and using Dotty as a dependency, which avoids fiddling with JARs manually:
Let dotty.scala contain:
//> using scala "3.3.1"
//> using dep "org.scala-lang::scala3-compiler:3.3.1"
@main def main(args: String*) = dotty.tools.dotc.Main.process(args.toArray)
Then we can run:
$ scala-cli --power package dotty.scala -o dotty-native --native-image --graalvm-args "--no-fallback"
This yields the same executable—which throws the same error—as the instructions in the issue description:
$ ./dotty-native -cp $SCALA_LIB hello.scala
Exception while compiling hello.scala
Exception in thread "main" java.lang.AssertionError: assertion failed: asTerm called on not-a-Term val <none>
at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
at dotty.tools.dotc.core.Symbols$Symbol.asTerm(Symbols.scala:169)
at dotty.tools.dotc.core.Definitions.ObjectClass(Definitions.scala:321)
...
We are one step further thanks to @sjrd!
A problem was apparently due to Dotty trying to read Java standard library modules, which did not work in native image.
We instead generated a JAR with Java standard library classes using a script in tasty-query (https://github.com/scalacenter/tasty-query/blob/main/build.sbt#L176-L206). This script can be run from SBT as follows:
$ git clone [email protected]:scalacenter/tasty-query.git
$ cd tasty-query
$ sbt "tastyQueryJS / Test / javalibEntry"
Using this JAR, we can now successfully compile hello.scala using the Dotty native image! 🥳
$ mkdir classes
$ time ./dotty-native -bootclasspath extracted-rt.jar -cp $SCALA_LIB -d classes hello.scala
./dotty-native -bootclasspath extracted-rt.jar -cp $SCALA_LIB -d classes 0.13s user 0.05s system 79% cpu 0.227 total
However, the generated class file seems to be corrupted:
$ java -cp $SCALA_LIB:$(pwd)/classes main
Error: Unable to initialize main class main
Caused by: java.lang.VerifyError: Constructor must call super() or this() before return
Exception Details:
Location:
main.<init>()V @0: return
Reason:
Error exists in the bytecode
Bytecode:
0000000: b1
Which is not the case if we generate it with the JVM version:
scala-cli dotty.scala -- -bootclasspath extracted-rt.jar -cp $SCALA_LIB -d classes-jvm hello.scala
My decompiler shows the same text for both:
// Source code is decompiled from a .class file using FernFlower decompiler.
import hello.package.;
import scala.util.CommandLineParser;
public final class main {
public main() {
}
public static void main(final String[] args) {
try {
.MODULE$.main();
} catch (CommandLineParser.ParseError var2) {
scala.util.CommandLineParser..MODULE$.showError(var2);
}
}
}
But the binaries of main.class are indeed different:
(File truncated, end not shown)
Hey @mbovel, thanks for digging into this! I'm not really familiar with Dotty, maybe you can help me understand the actual error a bit more. I assume asTerm() is supposed to be called in your app, only the assertion is not supposed to fail, right?
Exception in thread "main" java.lang.AssertionError: assertion failed: asTerm called on not-a-Term val <none>
at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
at dotty.tools.dotc.core.Symbols$Symbol.asTerm(Symbols.scala:171)
If Dotty or some other part are marked for build-time initialization, it's possible that "non-a-Term" object is created a build-time and frozen into the image. Maybe you could run with the -H:+PrintClassInitialization option for Native Image, which should provide you with an overview of all initialization policies.
Hey @mbovel ,
I noticed you were missing the config files from the scala-cli command. I got hello.scala compiled and run with this command:
scala-cli --power package dotty.scala -o dotty-native --native-image -f --jvm 17 -- --no-fallback -H:ConfigurationFileDirectories=native-image-config
