Incremental Compilation

[yuly16]

Add the incremental compilation support for scala native.

We set package as the granularity of the incremental compilation. In other words, in the edit-compile-run loop, we only generate the llvm code of the changed package. The llvm code of unchanged parts(e.g. most of the libraries and unchanged packages) won't be generated again.

We use the autotest for scala-native incremental compilation to test both the correctness and performance of incremental compilation. The benchmark refers to the scala native benchmark. The time cost of nativeLink is presented below. All binaries are tested to be correct.

1. Compiling in the first compilation(sbt clean + sbt nativeLink), the time cost is:

2. Compiling in later compilations(change a line in scala source code + sbt nativeLink), the time cost is:

Even if the incremental compilation adds some overhead in the first compilation, it brings significant performance improvement in later compilations. The overhead is caused by generating too many *.ll files. In the original version, the number of *.ll files equals to the number of cpu cores; when incremental compilation is applied, the number of *.ll files equals to the number of packages.

[ekrich]

It looks like there are some Java 11 methods being used like Path.of(String) that are not implemented. Also, maybe it would be a good idea to add a new option to NativeConfig for incremental with a default of true. What do you think @WojciechMazur ?

[ekrich]

It looks like there are some Java 11 methods being used like Path.of(String) that are not implemented. Also, maybe it would be a good idea to add a new option to NativeConfig for incremental with a default of true. What do you think @WojciechMazur ?

[lolgab]

This is great! Thank you very much! I have a question about how the split in .ll files happens. Why does the split happen per package? Have you considered an approach similar to what Scala.js does with SmallModulesFor(List("my-package")) where every other package not in the list is merged together (probably the same way as today) and you generate a file per class for my-package? Is it because of something specific to how Scala Native works?

[yuly16]

This is great! Thank you very much! I have a question about how the split in .ll files happens. Why does the split happen per package? Have you considered an approach similar to what Scala.js does with SmallModulesFor(List("my-package")) where every other package not in the list is merged together (probably the same way as today) and you generate a file per class for my-package? Is it because of something specific to how Scala Native works?

Hi, Lorenzo, the split happens per package because we set the granularity of the incremental compilation as package. For example, if we change somethings inside a Scala package, we need to recompile this Scala package to its corresponding .ll file. Previously we set the granularity as class instead of package, but we found that the overhead in the first compilation is too high when the granularity is class, because we have to generate too many .ll files in the first compilation. The more detailed reason is explained here: https://github.com/yuly16/Scala-Native-GSoC-Report

We didn't considered the approach similar to what Scala.js does, but I think it is a good idea to do incremental compilation. I believe Scala Native can do that.