Add support for Pipelined builds
This includes support for a single pass pipelined build, compatible with sbt's ThisBuild/usePipelining,
- adds
-Ypickle-javaand-Ypickle-writeflags, expected by Zinc when pipelining is enabled in sbt. - when
-Ypickle-write <directory|jar>is set, then write tasty from pickler to that output, (building upon https://github.com/lampepfl/dotty/pull/19074 support for Java signatures in TASTy files). - call
apiPhaseCompletedanddependencyPhaseCompletedcallbacks, which is the will activate early downstream compilation - calls
generatedNonLocalClasscallbacks early, which enables Zinc to run the incremental algorithm before starting downstream compilation (including checking for macro definitions).
generally this can be reviewed commit-by-commit, as they each do an isolated feature.
As well as many tests in the sbt-test/pipelining directory, this has also been tested locally on akka/akka-http, apache/incubator-pekko, lichess-org/lila, scalacenter/scaladex, typelevel/fs2, typelevel/http4s, typelevel/cats, slick/slick.
This PR sets the ground work for an optional 2-pass compile (reusing the OUTLINEattr), which should use a faster frontend (skipping rhs when possible) before producing tasty signatures
fixes #19743
We should probably add tests of incremental compilation with pipelining turned on, just to be sure there are no concurrency issues
TODO: sbt will write -Ypickle-java and -Ypickle-write twice in the Test scope (basically prepend to whatever was there), so we need to filter that out
We decided I should separate out the Java pickling ability as that is the only change that requires a minor release (changing TASTy Format), this will make the review much more simple to digest - then the rest of the features could come in patch releases (but I hope to still land this before 3.4.0)
edit: done in https://github.com/lampepfl/dotty/pull/19074
#19074 should be merged before this Edit: its merged :)
I need to rebase, which I aim to do today
I have a couple of usability bits to backport from my outlining branch (e.g. handle duplication of Ypickle-write and -Ypickle-java in test scope by Zinc), and probably move the pipeline jar writing to a separate (skippable) phase, then I can change the PR not to be draft
This PR is feature complete, there is one extra enhancement which is to asynchronously write the TASTy files to the early output - should it be included here?
Edit: @sjrd recommends to do that in a follow up PR.
I will not make be making any changes to this PR unless review demands it