bug
bug copied to clipboard
Scala 2.12.12 ArrayOps#++ no longer able to build GraalVM native image
This was reported originally as https://github.com/sbt/sbt/issues/5756
reproduction steps
using Scala 2.12.12:
val s = "foo"
val xs = s.getBytes("UTF-8") ++ System.lineSeparator.getBytes("UTF-8")
(actual code looks like https://github.com/sbt/sbt/blob/8ce423b088b85bb3016cfb994791c3536f7b627e/internal/util-logging/src/main/scala/sbt/internal/util/Terminal.scala#L384-L391)
Then build GraalVM native image.
problem
See https://ci.appveyor.com/project/sbt/sbt/builds/34671063/job/qveplsfqy7wyby3m
Error: Unsupported features in 2 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.MethodHandle.invokeBasic()
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The error is then reported at run time when the invoke is executed.
Trace:
at parsing java.lang.invoke.LambdaForm$MH/1971838936.invoke_MT(LambdaForm$MH)
Call path from entry point to java.lang.invoke.LambdaForm$MH/1971838936.invoke_MT(Object, Object):
at java.lang.invoke.LambdaForm$MH/1971838936.invoke_MT(LambdaForm$MH)
at scala.collection.immutable.VM.releaseFence(VM.java:25)
at scala.collection.immutable.HashSet$HashSetBuilder.result(HashSet.scala:1283)
at scala.collection.immutable.HashSet$HashSetBuilder.result(HashSet.scala:1192)
at scala.collection.TraversableLike.defaultPlusPlus$1(TraversableLike.scala:153)
at scala.collection.TraversableLike.$plus$plus(TraversableLike.scala:160)
at scala.collection.TraversableLike.$plus$plus$(TraversableLike.scala:147)
at scala.collection.mutable.ArrayOps$ofByte.$plus$plus(ArrayOps.scala:210)
at sbt.internal.util.Terminal$LinePrintStream.println(Terminal.scala:388)
at com.oracle.svm.jni.functions.JNIFunctions.ExceptionDescribe(JNIFunctions.java:759)
at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_ExceptionDescribe_b5412f7570bccae90b000bc37855f00408b2ad73(generated:0)
Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported type java.lang.invoke.MemberName is reachable: All methods from java.lang.invoke should have been replaced during image building.
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Trace:
at parsing java.lang.invoke.MethodHandles$Lookup.findVirtual(MethodHandles.java:861)
Call path from entry point to java.lang.invoke.MethodHandles$Lookup.findVirtual(Class, String, MethodType):
no path found from entry point to target method
com.oracle.svm.core.util.UserError$UserException: Unsupported features in 2 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.MethodHandle.invokeBasic()
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The error is then reported at run time when the invoke is executed.
Trace:
at parsing java.lang.invoke.LambdaForm$MH/1971838936.invoke_MT(LambdaForm$MH)
Call path from entry point to java.lang.invoke.LambdaForm$MH/1971838936.invoke_MT(Object, Object):
at java.lang.invoke.LambdaForm$MH/1971838936.invoke_MT(LambdaForm$MH)
at scala.collection.immutable.VM.releaseFence(VM.java:25)
at scala.collection.immutable.HashSet$HashSetBuilder.result(HashSet.scala:1283)
at scala.collection.immutable.HashSet$HashSetBuilder.result(HashSet.scala:1192)
at scala.collection.TraversableLike.defaultPlusPlus$1(TraversableLike.scala:153)
at scala.collection.TraversableLike.$plus$plus(TraversableLike.scala:160)
at scala.collection.TraversableLike.$plus$plus$(TraversableLike.scala:147)
at scala.collection.mutable.ArrayOps$ofByte.$plus$plus(ArrayOps.scala:210)
at sbt.internal.util.Terminal$LinePrintStream.println(Terminal.scala:388)
at com.oracle.svm.jni.functions.JNIFunctions.ExceptionDescribe(JNIFunctions.java:759)
at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_ExceptionDescribe_b5412f7570bccae90b000bc37855f00408b2ad73(generated:0)
Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported type java.lang.invoke.MemberName is reachable: All methods from java.lang.invoke should have been replaced during image building.
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Trace:
at parsing java.lang.invoke.MethodHandles$Lookup.findVirtual(MethodHandles.java:861)
Call path from entry point to java.lang.invoke.MethodHandles$Lookup.findVirtual(Class, String, MethodType):
no path found from entry point to target method
at com.oracle.svm.core.util.UserError.abort(UserError.java:79)
at com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:217)
at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:753)
at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:538)
at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:451)
at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
expectation
I can continue to build GraalVM native image?
note
The last call on the stack is scala.collection.immutable.HashSet.HashSetBuilder.result
: https://github.com/scala/scala/blob/v2.12.12/src/library/scala/collection/immutable/HashSet.scala#L1281-L1285
override def result(): HashSet[A] = {
rootNode = nullToEmpty(makeImmutable(rootNode))
VM.releaseFence()
rootNode
}
VM.releaseFence()
was added in https://github.com/scala/scala/pull/8722.
Related discussion happening in https://github.com/scala/bug/issues/11634.
I hit on the same issue and was able to work around it with the steps here https://github.com/scala/bug/issues/11634#issuecomment-670386111
I milestoned this 2.12.13 since in that timeframe perhaps someone™️ could at least document the situation
@olafurpg is there any reason your substitution code couldn't be the implementation in the repo? Do we want to avoid Unsafe maybe?
I'd love to not to have to make a change to work around.
We could publish a re-usable library that provides the native-image substitutions but I am not sure if it would work with any native-image version. Worst case, I suppose we could cross-build this library against multiple native image versions.
What about a Multi-Release JAR and using java.lang.invoke.VarHandle#releaseFence()
?
would it be possible to re-implement VM.releaseFence()
using an Indy instead of a MethodHandle
, to get around this issue but retain performance?
I published the workaround from https://github.com/scala/bug/issues/11634#issuecomment-670386111 as an independent library:
// Add this to build.sbt for the native-image project
libraryDependencies += "org.scalameta" %% "svm-subs" % "20.1.0" % "compile-internal"
The jar is 4kb and has no external dependencies besides scala-library. You can download the jar and add it manually to the classpath if you prefer https://repo1.maven.org/maven2/org/scalameta/svm-subs_2.13/19.3.2/svm-subs_2.13-19.3.2.jar
This workaround is only needed for 2.12.12+ and 2.13.3+
In case anyone is interested, I published a new plugin called sbt-native-image (https://github.com/scalameta/sbt-native-image) that automatically adds the correct svm-subs dependency and provides other nice features like automatic GraalVM installation
would it be possible to re-implement
VM.releaseFence()
using an Indy instead of aMethodHandle
, to get around this issue but retain performance?
I don't think so for two reasons:
- GraalVM does not support
invokedynamic
that change the method that is invoked. You would want to change the invoked method to callVarHandle
instead ofUnsafe
on Java 9+. - I'm note aware of a way to have control over
invokedynamic
from Scala source.
Good news:
When everything is finished, all aspects of method handles including bootstrap methods will be supported.
(from https://github.com/oracle/graal/issues/2761#issuecomment-701531195)