scala3 icon indicating copy to clipboard operation
scala3 copied to clipboard

Add support for Pipelined builds

Open bishabosha opened this issue 2 years ago • 6 comments

This includes support for a single pass pipelined build, compatible with sbt's ThisBuild/usePipelining,

  • adds -Ypickle-java and -Ypickle-write flags, expected by Zinc when pipelining is enabled in sbt.
  • when -Ypickle-write <directory|jar> is set, then write tasty from pickler to that output, (building upon https://github.com/lampepfl/dotty/pull/19074 support for Java signatures in TASTy files).
  • call apiPhaseCompleted and dependencyPhaseCompleted callbacks, which is the will activate early downstream compilation
  • calls generatedNonLocalClass callbacks early, which enables Zinc to run the incremental algorithm before starting downstream compilation (including checking for macro definitions).

generally this can be reviewed commit-by-commit, as they each do an isolated feature.

As well as many tests in the sbt-test/pipelining directory, this has also been tested locally on akka/akka-http, apache/incubator-pekko, lichess-org/lila, scalacenter/scaladex, typelevel/fs2, typelevel/http4s, typelevel/cats, slick/slick.

This PR sets the ground work for an optional 2-pass compile (reusing the OUTLINEattr), which should use a faster frontend (skipping rhs when possible) before producing tasty signatures

fixes #19743

bishabosha avatar Nov 08 '23 15:11 bishabosha

We should probably add tests of incremental compilation with pipelining turned on, just to be sure there are no concurrency issues

bishabosha avatar Nov 11 '23 11:11 bishabosha

TODO: sbt will write -Ypickle-java and -Ypickle-write twice in the Test scope (basically prepend to whatever was there), so we need to filter that out

bishabosha avatar Nov 13 '23 16:11 bishabosha

We decided I should separate out the Java pickling ability as that is the only change that requires a minor release (changing TASTy Format), this will make the review much more simple to digest - then the rest of the features could come in patch releases (but I hope to still land this before 3.4.0)

edit: done in https://github.com/lampepfl/dotty/pull/19074

bishabosha avatar Nov 23 '23 09:11 bishabosha

#19074 should be merged before this Edit: its merged :)

bishabosha avatar Nov 24 '23 16:11 bishabosha

I need to rebase, which I aim to do today

bishabosha avatar Feb 09 '24 14:02 bishabosha

I have a couple of usability bits to backport from my outlining branch (e.g. handle duplication of Ypickle-write and -Ypickle-java in test scope by Zinc), and probably move the pipeline jar writing to a separate (skippable) phase, then I can change the PR not to be draft

bishabosha avatar Feb 12 '24 17:02 bishabosha

This PR is feature complete, there is one extra enhancement which is to asynchronously write the TASTy files to the early output - should it be included here?

Edit: @sjrd recommends to do that in a follow up PR.

bishabosha avatar Mar 19 '24 08:03 bishabosha

I will not make be making any changes to this PR unless review demands it

bishabosha avatar Mar 25 '24 12:03 bishabosha