Add experimental Kotlin incremental compilation

[erikkerber]

[!NOTE] This is presented as a POC, but nowhere near merge-ready. In its current state, it may compile your apps and you may see significant build time improvements.

This adds an experimental implementation of incremental compilation via the new "Build Tools API" (BTAPI). The BTAPI itself can replace the direct usage of K2JVMCompiler, but is also required to leverage incremental compilation.

Initial Design

In short:

• Create an alternate KotlinCli wrapper to BazelK2JVMCompiler named BuildToolsApiCompiler. This invokes the BTAPI.
• Store incremental kotlin artifacts (unsafely) in the execution root (i.e. <output base>/execroot/_kotlin_incremental/)
• (Temporary) Generate all classpath snapshots in-situ before the build. See TODO item.
• Add two toolchain flags:
  • build_tools_api to enable building with the new BTAPI (only)
  • incremental_compilation to then add incremental compilation

The rest is fairly trivial usage of the BTAPI, a little of which requires some tinkering and tribal knowledge to know how exactly it is supposed to be used. I did my best to document my knowledge, with brevity.

Performance

The main driver of performance gains are incremental builds that only build a subset of Kotlin files in a target. The incremental data passed in to subsequent builds gives an additional level of work avoidance beyond the action-based caching in Bazel.

I hope to get more comprehensive numbers, but our main test case was a sample incremental build that was ~4x faster in Gradle due to only needing to recompile very small slices of a chain of Kotlin modules. With incremental, Bazel matches what we see with Gradle.

4x isn't a fixed number, it could be greater or smaller depending on how many Kotlin files are in how many dependent targets, and how many files can skip compilation based on the affected classpaths in a build.

TODO

• [ ] Track classpath snapshots in Bazel as inputs/outputs
• [ ] Solve for BTAPI dependency w/Kotlin toolchain setup and versioning. Currently it is only distributed via Maven.
• [ ] Settle on a strategy for calculating Kotlin incremental directories. Currently I'm creating isolated directories based on label + metadata passed from CompilationTask. It works, but not idea.
• [ ] Migrate to the new BTAPI via https://github.com/Kotlin/KEEP/blob/build-tools-api/proposals/extensions/build-tools-api.md
• [ ] Tests, everywhere