Support features of JDK 11+

[retronym]

In general, Scala now works just fine on JDK 11

For the current support status of each JDK version, see https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html

JDK9 JEP list

• JEP 223: New Version-String Scheme
  • [x] Update our version parsing in isJavaAtLeast (and elsewhere?) https://github.com/scala/scala/pull/5276
• JEP 220: Modular Run-Time Images (includes SI-9103)
  • [x] Adapt our classpath to use jrt:/ virtual filesystem. WIP retronym:review/5112
• [ ] Update Zinc to understand that jrt:// classpath entries should have a timestamp based on the JDK (maybe $JAVA_HOME/jmods/*.mod? Find out exactly what file backs jrt://)
• [ ] Adapt REPL :javap command (currently gets IllegalArgumentException in JavapClass$JavapTool$.apply(JavapClass.scala:337)
• JEP-261 Module System (https://github.com/scala/scala-dev/issues/529)
  • [ ] javac allows programmers to declare module requirements, exports inline in the sourcetree ("module-info.java" files). What should we do?
  • [ ] app class loader is no longer a URLClassloader. Lots of places enumerate the classpath for JARs to initialize settings for a compiler. Experiment, document.
  • [ ] Update classfile reader to parse module-info.class (perhaps using ASM)
  • [ ] create module readability / accessibilty graph (following transitive "public exports")
• JEP-238 Multi-Release JAR Files
  • [x] See if this feature is used in practice
  • [x] Update classpath implementation to be able to consume such JARs. (WIP)
• JEP 280: Indify String Concatenation
  • [x] Under target:jvm-1.9, consider emitting Javac style string concat with indy.
    • https://github.com/scala/scala/pull/9556
  • [x] Update to ASM 5.1 (?). As suggested by @varming https://github.com/scala/scala-asm-legacy/pull/11
• [ ] JEP-193 VarHandles introduce a number of new methods annotated with @PolymorphicSignature. Expand our compiler support to these methods.
  • [ ] Maybe: foster creation of a library of Java 8/9 shims that use invokedynamic to use some Java 9 features (e.g VarHandle) when available but fall back to Java 8 alternatives (e.g. Unsafe._), as prototyped in https://github.com/retronym/scala/commit/a95f03e8b86c1c28f4a86346697ae0ff5629dfc4
• Infrastructure:
  • [x] Make JDK9 EA builds available in CI: https://github.com/scala/scala-jenkins-infra/issues/174
• SBT
  • [ ] Review Zinc's use of sun.boot.classpath: https://github.com/sbt/zinc/blob/1.x/internal/zinc-compile-core/src/main/scala/sbt/internal/inc/CompilerArguments.scala#L83
• Make mixin fields non-final to avoid verify errors. Use explicit fences to preserve memory effects: #408

[retronym]

To whet the collective appetite, here's my branch that adapts to JEP 220 up and running:

⚡ qscala
Welcome to Scala 2.12.0-20160425-151134-5e07d57 (Java HotSpot(TM) 64-Bit Server VM, Java 9-ea).
Type in expressions for evaluation. Or try :help.

scala> java.util.List.of(1, 2, 3)
res0: java.util.List[Int] = [1, 2, 3]

[varming]

With less than nine months to the release of JDK9 not being able to run Scala programs on the early access releases is a serious concern, so it is great to see that work has started.

[SethTisue]

@retronym did 2.12.1 check some more of these boxes?

[retronym]

I have submitted a PR to SBT to start the ball rolling for Java 9 support. Because SBT internally uses an old version of scalac to compile build definitions, I decided to create a tool to export a jrt:/modules/*/** to rt.jar to bridge JDK9 to the older compiler.

This approach also enables your project to be compiled with, say, Scala 2.11.8, which might help some more folks test out JDK9, without first needing to upgrade to the 2.12 series.

Warning: we are yet to implement the additional access checks required by JEP-261. Until then, you can get LinkageError-s at runtime for code that passes the Scala type checker. So while I encourage testing and experimentation, use this at your own risk in production.

[retronym]

Discussion on the Java Language Spec changes related to modules, thread started by Eclipse JDT seeking to implement the spec. http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-May/012433.html

[ijuma]

Another Java 9 issue: https://github.com/scala/bug/issues/10418

[SethTisue]

I've created https://github.com/scala/bug/labels/jdk9

[ijuma]

JEP 247: Compile for Older Platform Versions is also relevant. It would be nice if scalac could understand $JDK_ROOT/lib/ct.sym so that a newer JDK could be safely used to build for an older version.

[asarkar]

Is there a target release for addressing all the issues listed here, and reported by the users, or is it going to be on best-efforts basis?

[retronym]

@asarkar My primary goal at the moment is removing impediments to building and running Scala 2.12 on JDK9. We have been able to build and test the Play Framework, and the Akka team is working through similar testing at the moment.

I then want to implement support in the Scala typechecker for JPMS. I expect this will be done in Q4 this year. I'm hopeful that we could ship this in either a minor release of Scala 2.12, or as an experimental compiler plugin for Scala 2.12. It will introduce a dependency on the Java 9 standard library, which will complicate our build somewhat, as in general the compiler should only require Java 8.

I don't have a timeframe for the full list, but I'm keen to hear about what parts are most important for our users to help prioritize.

[smarter]

Update Zinc to understand that jrt:// classpath entries should have a timestamp based on the JDK (maybe $JAVA_HOME/jmods/*.mod? Find out exactly what file backs jrt://)

Surely that's not right for modules on the modulepath that are not part of the JDK?

[retronym]

Once we have a --modulepath compiler option, we'll be able to access the contents are regular JARs (well, apart from the .jmod format). So we might not actually access them via the jrt:// filesystem at all.

[lukaseder]

I managed to compile stuff on the JDK 9 using Scala 2.12.3, but not using 2.11.11 where I'm getting this error:

error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.

Is JDK 9 usage not supported with 2.11, or might there be a bug worth reporting?

[lrytz]

This is probably https://github.com/scala/scala-dev/issues/304, fixed for 2.12 in https://github.com/scala/scala/pull/6097. We're backporting some fixes for a 2.11.12 release, should be out soon.

[lukaseder]

Cook, thanks very much for the pointer, @lrytz !

[mkurz]

I opened pull request https://github.com/scala/scala/pull/6146 to add support for -target:jvm-1.9.

[clhodapp]

It would be great if Scala Team could publish an official position on the readiness of Scala for JDK/JRE9 (even if it's just a line or two integrated into the download directions on scala-lang.org). As it stands, it's not clear at all to most developers whether it's considered safe to use Scala with Java 9. This is, of course, completely separate from fully leveraging Java 9, which is obviously going to be a substantial ongoing effort for quite a while (as reflected here).

[SethTisue]

re: download directions on scala-lang.org, see https://github.com/scala/scala-lang/issues/761

[markehammons]

I would like to say

javac allows programmers to declare module requirements, exports inline in the sourcetree ("module-info.java" files). What should we do?

is of great importance to me. While I've been running my scala code on java 9 recently (and also openJ9), I haven't bothered messing with modules yet because scala doesn't support them. This is a shame imo, cause while stuff like private[package] is nice, I'd really love to have jpms modules instead.

[retronym]

@markehammons The plan is that you will define your module in the same way as you would in Java by adding a module-info.java file to your source tree. All the build tools support mixed Java/Scala projects.

The missing piece of the puzzle is getting scalac to respect these limitations.

I've just finished backporting the of Java 9 support we have so far from 2.12.4 back to the imminent 2.10.7 and 2.11.12 to help folks migrate. In particular, this is designed to help people using SBT 0.13 (which implies use of Scala 2.10.x).

Once those releases are done, I'll add support in 2.12.5 for respecting multi-release JARs and the --release compiler option (to limit compilation to, e.g. the Java 8 subset of the Java standard library). These are relatively low hanging fruit.

Next cab off the rank will be full support for JPMS modules.

@clhodapp After the backport releases I'll blog about the state of JDK 9 support and the roadmap.

[ijuma]

Happy to hear about --release. As you say, it's low hanging fruit and it simplifies the contributor experience (no need to ask them to install an older JDK).

[schmitch]

@retronym so the way forward to use the module-system is to place a module-info.java into the scala/ folder? doesn't that seems wierd. Can't the scala compiler have his own module-info.scala, I don't think it would be great to put *.java files in 100% scala libraries?!

[ktoso]

If it would be a *.scala file then one would expect not having to put ; etc, so we'd have to have our own parser perhaps for it, making is slower to adopt it (and more potential for diverging from the original). I personally don't mind the module-info.java files -- it's more of a "the java runtime cares about this" anyway and since it's a form of DSL anyway. One could add them to java/ usually I think anyway.

[schmitch]

well but wouldn't that mean that scala-js/scala-native needs to ingore any module-info.java?

[ktoso]

Indeed, since it's a java runtime thing. What about it?

[saosebastiao]

javac allows programmers to declare module requirements, exports inline in the sourcetree ("module-info.java" files). What should we do?

A suggestion: modify or make available several different App-like trait mixins which provide implicit values representing module capabilities. Then you statically control availability of the resulting classes and methods via implicit views or generalized type constraints. For example:

object Main extends JVM9ModuleApp with ConsoleModule

would make System.out.* available via a dependency on the relevant jdk module, and it would expose methods like println via a (implicit console: C =:= Console) type constraint. If ConsoleModule is not mixed in, the implicit resolution would fail and println would generate a compile time error.

Just as an FYI, this approach would be considered a form of a coeffect system. Coeffects are the dual of effects: where an effect tracks what a program does to the world, a coeffect tracks what is required of it. It could also be extremely extensible...if the implicit coeffect provided represents availability of a type of code generator in the compiler, it could dispatch that code to that code generator. This would allow true unification of the available backends (scala-native, scalajs, scalajvm) into a single compiler, and allow cross compiled apps as well as typesafe client-server apps within the same compilation project. It could also enable non-jvm targets to finely control code stdlib availability as well as code generation, taking into consideration the os/platform APIs that are available. Some possibilities include:

object Client extends JsApp with ES6Gen { ... }
object Server extends JVM8App with CudaGen { ... }` 

or even

object MyAndroidApp extends AndroidApp with API26 with NetworkPermissions with LocationPermissions { ... }
object MyIOSApp extends NativeApp with MulticoreRuntime with CocoaTouch { ... } 

Tomas Petricek of the F# community has done a lot of work formalizing the concept of coeffects and has created an amazing website dedicated to explaining the topic. The possibilities are enough to make me giddy. I love scala as a language, and I love the direction that the community is going with it (scalajs, scala-native, etc.), but all of that unified via a coeffect system could be transformative for cross-platform programming in the same way that Rust is transforming systems programming. And the jdk9 module system is the perfect start point for it.

[devshorts]

Out of curiosity is there still activity on this? I realize this is a tricky ticket, but JDK10 is scheduled for GA in march http://openjdk.java.net/projects/jdk/10/ so it'd be nice to not be 2 major versions behind in scala land

[smarter]

@devshorts This ticket covers a wide range of Java 9 related things. It'd be helpful if you could indicate which ones in particular you're interested in. Note that the latest releases of sbt, Scala 2.10, 2.11 and 2.12 work correctly on Java 9 and produce code that can be run on Java 9.

[SethTisue]

@devshorts JDK 9 in general is still very much on the collective mind of the Scala and tooling teams at Lightbend and remains near the top of our priorities. (and thanks @smarter for noting some of the progress made in the last few months)

[SethTisue]

note that community participation in making progress on remaining checkboxes and helping them happen faster is very much welcome