Ongoing Scala 3 migration effort

[jrudolph]

Projects can already use Akka HTTP with Scala 3 by using CrossVersion.for3Use2_13, but for libraries that is not a good solution. This PR is about providing 'native' Scala 3 artifacts for Akka HTTP.

Commits are supposed to stay, so let's try to use reasonable commits. Changes can be PR'd against this branch.

Here are some guidelines about what we want to achieve:

• Scala 3 support for all modules
• No significant code forks (version-dependent directories) of general purpose code. We can maintain some forks (e.g. in parsers if not possible otherwise) but in general we cannot maintain significant forks for Scala 3 compatibility. I.e. for now we need to find solutions that work for all of Scala 2.12, 2.13, and 3. If Scala 2.12 support is too big a burden, we might consider targeting Akka HTTP 10.3.0 where we might want to drop Scala 2.12 support (and Akka 2.5).
• Only Akka 2.6 is supported, so I changed the default Akka version. Since we still build against Akka 2.5 by default we will have to find a solution for publishing the Scala 3 version against 2.6.
• Please add a comment when working on a task item to avoid stepping on each other's toes.

Tasks:

• [x] Port parboiled2 to our private fork
• [x] akka-http-core
  • [x] Make main sources compile without parsers
  • [x] Gradually make parsers compile for Scala 3 #4097
  • [x] Incrementally move tests back from scala-2 to scala directory, fixing issues in the process #4097
  • [x] Setup Scala 3 compiler plugin to elide methods marked with @pre213 annotation (or wait until Scala 2.12 is dropped) #4113
• [x] akka-http2-support (only has tests)
• [x] akka-http
  • [x] Enable akka-http module for Scala 3 (currently disabled in Scala 3) and make main code compile @jchyb
• [x] akka-http-testkit @jrudolph #4101
• [x] akka-http-tests #4107
  • [x] Make tests compile #4107
  • [x] Make tests pass #4107
• [x] Enable akka-caching @jrudolph #4101
• [x] Enable akka-http-spray-json #4107
• [x] Enable akka-http-scala-xml #4107
• [x] Enable akka-http-jackson #4126
• [ ] akka-http-bench-jmh
  • [ ] Run some benchmarks to find most obvious issues
• [x] docs module #4126
• [x] Make compilation work without 3.0-migration compiler option https://github.com/akka/akka-http/pull/4114
• [ ] Backport non-intrusive changes to main (or whatever we call the 10.2.x branch) so we minimize the diff between 10.2.x and 10.3.x for easier backporting later on

For now start sbt with sbt -Dakka.build.akkaVersion=2.6.18 ++3.1.1 shell

[jrudolph]

If all parser sources are enabled, Scala 3 still gets into an unlimited loop. A relatively simple (though, nested) Parser that shows this behavior is IpAddressParsing.

[jchyb]

Hello, I started working on the "Enable akka-http module for Scala 3 (currently disabled in Scala 3) and make main code compile" task, or at least as much as can be done without all of the akka-http-core parsers. Good luck everyone!

[jrudolph]

Hello, I started working on the "Enable akka-http module for Scala 3 (currently disabled in Scala 3) and make main code compile" task, or at least as much as can be done without all of the akka-http-core parsers. Good luck everyone!

Great, thanks!

[prolativ]

I started working on the parsers. At least for now I have found out that that there's no infinite loop but the compilation time grows very quickly when a rule gets just slightly more complicated

[hughsimpson]

I've been noodling around with akka-http-core and found that if there were more than two non-literal subrules in a parser chained with ~ it basically takes forever to compile, unless you force right-associativity with parentheses (i.e. `ip-number` ~ ('.' ~ (`ip-number` ~ ('.' ~ `ip-number`))) compiles fast enough, but `ip-number` ~ '.' ~ `ip-number` ~ '.' ~ `ip-number` basically takes forever). That trick allows pretty much all the rules to compile before one dies of old age, except for a couple of the top-level date parsers where it's still taking about 10 mins on my machine, and the UriParser stuff, which I didn't look at yet. Might be able to tidy up and pr some of that this week if all goes well. Not sure what the underlying cause for that slowdown is, though...

[jrudolph]

I've been noodling around with akka-http-core and found that if there were more than two non-literal subrules in a parser chained with ~ it basically takes forever to compile

Interesting, thanks for trying! Maybe that's something that we can investigate in isolation directly in parboiled2.

[jchyb]

I started working on making akka-http-tests tests compile. That required re-enabling akka-caching, akka-http-spray-json, akka-http-scala-xml, akka-http-jackson and akka-http-testkit, all of which were mostly simple - the only real choice to be made there was updating the version of Specs2, for now I updated it from 4.10.6 to 4.15.0, which is cross-compiled to Scala 2.12, 2.13 and 3

[hughsimpson]

@jrudolph I have a workaround for the compilation times that's I think is 'good enough' for Akka-http; have pr'd to this branch to demonstrate it. I'll pr to parboiled2 later today

EDIT: parboiled2 pr https://github.com/sirthias/parboiled2/pull/362

[jchyb]

I have managed to compile and run akka-http-tests without the 3.0-migration flag - but I think I should wait for #4107 to be merged before making another PR for tests. I also did the same thing for akka-http-core - curiously, while working on that I discovered that when compiling with 3.1.3-RC2, some new issues appear there connected to the parsers (type inference, rather then long compilation times like before), but I have not investigated them yet

[jrudolph]

curiously, while working on that I discovered that when compiling with 3.1.3-RC2, some new issues appear there connected to the parsers (type inference, rather then long compilation times like before), but I have not investigated them yet

Yes, I saw this as well and filed https://github.com/sirthias/parboiled2/issues/365. Not sure what to do about that. Either, we need to stay on 3.1.2 for the time being or someone needs to invest even more time to figure out how to fix that...

[jchyb]

I have setup a compiler plugin like in the Setup Scala 3 compiler plugin to elide methods marked with @pre213 annotation (or wait until Scala 2.12 is dropped) task, and It does not seem to work too well. When @pre213 are uncommented, we still get a bunch of DoubleDefinition errors - looking at dotty source, they seem to come from the typer phase. I've tried setting up so that the plugin is run before it, but then the compiler ignores it altogether. According to the documentation, it would be possible to setup a ResearchPlugin with that purpose, but those are heavily limited, being allowed only for nightly language releases. Unfortunately, I don't think compiler plugin way of eliding is possible here

[jrudolph]

I have setup a compiler plugin like in the Setup Scala 3 compiler plugin to elide methods marked with @pre213 annotation (or wait until Scala 2.12 is dropped) task,

Interesting, could you push your attempt somewhere?

[jrudolph]

curiously, while working on that I discovered that when compiling with 3.1.3-RC2, some new issues appear there connected to the parsers (type inference, rather then long compilation times like before), but I have not investigated them yet

Yes, I saw this as well and filed sirthias/parboiled2#365. Not sure what to do about that. Either, we need to stay on 3.1.2 for the time being or someone needs to invest even more time to figure out how to fix that...

This has now been fixed in https://github.com/lampepfl/dotty/pull/14987. For now 3.2.0-RC1-bin-20220424-0761c50-NIGHTLY is a nightly that works again. A backport for 3.1.3 has also been merged but there don't seem to be nightlies available.

[jrudolph]

I have setup a compiler plugin like in the Setup Scala 3 compiler plugin to elide methods marked with @pre213 annotation (or wait until Scala 2.12 is dropped) task,

Ok, I also had a quick attempt but this not going to fly, indeed. I'll have a quick try if we can just edit the source files on the fly in sbt. If that doesn't work, we go with version-dependent files.

[jrudolph]

I'll have a quick try if we can just edit the source files on the fly in sbt

Seems simple enough https://github.com/akka/akka-http/pull/4113

[hughsimpson]

Is anyone doing akka-http-jackson? I might be able to have a look tomorrow

[jrudolph]

Just to let you know: we are not yet sure how to release this branch. Supporting Scala 2.12, 2.13, and 3 from the same branch is somewhat of a challenge and we wonder if it makes sense if we need to support all of those together for quite a while (even if Scala 3 support is not officially offered by Lightbend right now, it will, of course, still be part of the codebase etc.).

So, it might be that we we'll plan for a 10.3.x later this year where we drop Scala 2.12 and Akka 2.5 support, and add Scala 3 support. When we have that finally decided, we can probably relatively quickly go for 10.3 milestones to get releases with Scala 3 out of the door.

Any issues with that tentative plan?

[jrudolph]

To clarify, this branch would then become the start of the 10.3.x release branch.

[mkurz]

No issues with that. It would be nice to have at least milestone releases with Scala 3 asap, so we can use them in Play. Thanks!

[godenji]

Just to let you know: we are not yet sure how to release this branch. Supporting Scala 2.12, 2.13, and 3 from the same branch is somewhat of a challenge and we wonder if it makes sense if we need to support all of those together for quite a while (even if Scala 3 support is not officially offered by Lightbend right now, it will, of course, still be part of the codebase etc.).

So, it might be that we we'll plan for a 10.3.x later this year where we drop Scala 2.12 and Akka 2.5 support, and add Scala 3 support. When we have that finally decided, we can probably relatively quickly go for 10.3 milestones to get releases with Scala 3 out of the door.

Any issues with that tentative plan?

I must be missing something. What was the point of this rather substantial migration effort if right on the finish line there will in fact be no Scala 3 releases until end of year 2022/start of 2023? How are projects like Play supposed to work their way toward integrating Scala 3 support? How are Akka HTTP users supposed to try out Scala 3 based artifacts?

I don't mean to be critical, just trying to understand the rationale for such a lengthy delay -- why not just include Scala 3 in the mix with the caveat that Lightbend does not yet officially support it, YMMV, or have some build flag that omits Scala 3 from stable Akka HTTP releases? (i.e. only include Scala 3 in snapshot/nightly releases).

[raboof]

What was the point of this rather substantial migration effort if right on the finish line there will in fact be no Scala 3 releases until end of year 2022/start of 2023?

Even if a release would be some way off, we do plan to start publishing nightly snapshots as soon as possible.

How are projects like Play supposed to work their way toward integrating Scala 3 support?

Play could then hopefully already start using those snapshots for their Scala 3 work.

why not just include Scala 3 in the mix with the caveat that Lightbend does not yet officially support it, YMMV, or have some build flag that omits Scala 3 from stable Akka HTTP releases? (i.e. only include Scala 3 in snapshot/nightly releases).

'Just' including Scala 3 in the mix for the 10.2.x branch is challenging because that means we'd have to support Scala 2.12, 2.13, and 3 in the same codebase. Dropping Scala 2.12 would make it easier to get Scala 3 support in, but we don't want to drop Scala 2.12 support within 10.2.x, so that'd need to be on a 10.3.x branch. Of course as soon as we have that in a working state we'd start publishing nightly snapshots for that branch as well.

[raboof]

Also "later this year" does not necessarily mean "end of year 2022/start of 2023" ;)

[godenji]

Also "later this year" does not necessarily mean "end of year 2022/start of 2023" ;)

That's good news, thanks for the clarification.

[strelec]

Hey @jrudolph,

Even though the support for Scala 3 is months away, I suggest you commit low risk pre-work that does not affect scala 2. For example https://github.com/akka/akka-http/pull/4079/commits/979ac0c1d5cc7dc8dc7db96c9e7839a62a4c264b and https://github.com/akka/akka-http/pull/4079/commits/0135521723cc6d2599b909e771f56c6a947719cc.

Why?

1. Less chance for merge conflicts when it's time
2. You will have parts of your pull request already validated for correctness by production usage

[mkurz]

Hi! Will this Scala 3 support make it into akka-http 10.2.x? Or will there be a new major version 10.3.x when akka 2.7.x will be released?

[lightbend-cla-validator]

Hi @jrudolph, @raboof, @jchyb, @hughsimpson,

Thank you for your contribution! We really value the time you've taken to put this together.

We see that you have signed the Lightbend Contributors License Agreement before, however, the CLA has changed since you last signed it. Please review the new CLA and sign it before we proceed with reviewing this pull request:

https://www.lightbend.com/contribute/cla

[lightbend-cla-validator]

Hi @jrudolph, @raboof, @jchyb,

Thank you for your contribution! We really value the time you've taken to put this together.

We see that you have signed the Lightbend Contributors License Agreement before, however, the CLA has changed since you last signed it. Please review the new CLA and sign it before we proceed with reviewing this pull request:

https://www.lightbend.com/contribute/cla

[mushtaq]

@jrudolph Any way of trying out akka-http snapshot artifacts for scala-3?

[lightbend-cla-validator]

Hi @jrudolph, @raboof, @jchyb,

Thank you for your contribution! We really value the time you've taken to put this together.

We see that you have signed the Lightbend Contributors License Agreement before, however, the CLA has changed since you last signed it. Please review the new CLA and sign it before we proceed with reviewing this pull request:

https://www.lightbend.com/contribute/cla

[johanandren]

Not yet, for now you would have to build your own snapshot out of this branch.