Path to Scala 3

[felixbr]

This is an issue for discussing and tracking necessary step to prepare for a Dotty/Scala3 cross-build.

Inspired by the awesome "Scala Love" conference, I attempted to cross-build util-core with Dotty 0.23.0-RC1. So far it went smoother than expected. In the end there were about 4 compile-errors that I couldn't directly address.

In any case there's some work that has to be done before a real cross-build can be achieved. So far I've identified the following but I will try and update this list as we learn more.

Preparations:

• [x] Update sbt to recent version (e.g. 1.3.10; needed by sbt-dotty)
• [x] Update Scalatest to 3.1.1 (which is cross-published for Dotty)
• [x] Update twitter/util to Scalatest version available for latest Dotty (Dependency on Twitter codebase) | @mosesn
• [x] Remove deprecated Config code | @mosesn
• [x] Wait for Dotty to fix varargs override issue (see https://github.com/lampepfl/dotty/issues/9463)
• [x] Wait for Scalatest 3.2.0 to be published for Dotty version which includes the varargs fix
• [ ] Refactor code which is not supported in Scala 3 (e.g. usage of Manifest/TypeTag reflection)
• [ ] Add Scala 3 to the CI matrix to prevent regressions in already supported modules
• [ ] Cross-build and cross-publish modules which are needed by Finagle and twitter-server (e.g. util-test and util-tunable)

Resources for Dotty migration/cross-building:

• https://github.com/scalacenter/scala-3-migration-guide
• https://github.com/lampepfl/dotty-example-project#getting-your-project-to-compile-with-dotty

I hope you don't mind this type of issue. This is not meant to push anyone, just as a place to discuss and organize things.

Cheers ~ Felix

[mosesn]

@felixbr yep, we're perfectly happy to have those kinds of conversations here. If it turns into a big effort, we'll probably set up a ticket like https://github.com/twitter/finagle/issues/771 for doing it. We're going to upgrade scalatest soon to support https://github.com/twitter/finatra/pull/527.

I can file an internal ticket around upgrading our sbt version, but we're going through some turbulence with our builds, so there's a good chance that we'll have to hold off on that until we've worked out some of the details.

[mosesn]

Oh, also, as we find things that are challenging to migrate, that's probably good feedback for the scala compiler team. Especially if there are things that scalafix handles poorly. Please let us know what your experience in trying it out is like! Thanks for taking this on, I'm excited to see what you find =)

[cacoco]

@felixbr FYI, we've updated to sbt 1.3.10 in 6ee7b49a7aa22462648dd9be27e2c577b3a9f003. Scalatest upgrade may take a bit longer. Thanks.

[mosesn]

@felixbr sorry for the delay, scalatest was a beast. It's finally updated https://github.com/twitter/util/commit/db63a6dcf2113606f27e04d6860ad2d0ca72ab5e

[felixbr]

Oh, that's great news! :)

[mosesn]

@felixbr what do you think are the next steps to supporting scala 3? We've tackled the two you suggested before.

[felixbr]

The biggest problem is that Dotty/Scala 3 is not stable yet, so (if I understand correctly) for any given Dotty version we want to use for a cross-build, the dependencies have to be available. It's a moving target and that makes it difficult at the moment.

I just looked at maven-central and it seems that for Dotty 0.23 a Scalatest 3.1.2 version and for Dotty 0.24 a Scalatest 3.2.0 version exists. The latest Dotty version is 0.25 (released 5 days ago).

My guess is that moving to Scalatest 3.2.0 could be a lot of work again, so for the moment we could attempt the cross-build with Dotty 0.23 but I have no idea how outdated this is compared to 0.25.

Maybe that's something we should clarify with the Dotty-Team before we attempt cross-building with an older version.

[mosesn]

@felixbr I'm inclined to just start with 0.23, and we can move to newer versions after we upgrade our scalatest. I think scalatest 3.1.x's scalafix rewrite will be easier to use than the 3.0.8 one, so I'm cautiously optimistic that the 3.2.0 upgrade won't take that long.

@SethTisue what do you think is the right way forward here?

[mosesn]

Poking around a bit, it looks like @martijnhoekstra also tried porting util to dotty, and ran into some issues: https://contributors.scala-lang.org/t/scala-3-migration-guide/4237/6

What is the usual strategy that people take? Do they keep two different sets of code, or do they use the subset of scala that is supported by both scala 2 and scala 3? I think one approach I would be happy with would be to keep developing in scala 2, and then have a tag that continuously tracks "develop but scala-3-ified"

[martijnhoekstra]

@mosesn to me it made sense at the time to report the encountered issues to dotty and not bother with workarounds yet, since introducing (sometimes ugly) workarounds for bugs that probably won't be in scala 3.0 release didn't make sense to me.

IIRC the biggest issue was the dropped support for macros which is used in IIRC the deprecated Configuration stuff (with existing notes about breakages). Apart from that, cross-building with dotty seems plausible on a unified codebase without any scala-2/scala-3 specific stuff.

Until scala 3.1 rolls along that is, but it seems too early to worry excessively about that.

[mosesn]

Cool! I'm glad to hear it. Which Configuration stuff? Util doesn't have any macros afaik

[martijnhoekstra]

I took another look @mosesn

It wasn't macros, but it was a dependency on scala-reflect in Config (filtering out synthetics).

Apart from that, for now the dealbreakers look like the same as mentioned on the contributors thread: no @varargs support just yet (so no java-compatible varargs forwarders and no overriding of java-compatible varargs, needed in InMemoryStatCounter) and no good story for cross-compiling dynamic member selection (on unsafe in jvm/HotSpot)

[martijnhoekstra]

https://github.com/twitter/util/compare/develop...martijnhoekstra:dotty-compat is a quick port -- in this state all main projects compile (but test crash-and-burns)

[mosesn]

Got it. What do you think would be the right way to interact with the dotty team to get them to prioritize this stuff? It would be really neat for us to be able to add util to the dotty community build.

[felixbr]

@martijnhoekstra I found pretty much the same blockers when I attempted the cross-build. I didn't really understand what exactly scala-reflect was used for. Do you think we can just remove it?

Maybe we should gather the missing features and then talk to the Dotty team, which of them can be expected to be solved by them and which need a separate workaround anyway.

• [ ] Dependency on scala-reflect for "synthetic term names" (?)
• [ ] Java-compatible varargs support
• [ ] Dependency on scalactic.source.Position macro for logging
• [ ] "dynamic member selection" (?)

Are these all blockers right now (apart from tests, of course)?

In general I'm in favor of trying a unified codebase first even though writing code that works on 2.11 through 3.0 will be interesting :smile:

[martijnhoekstra]

I bet it would be neat for the dotty team too, I think 2.13 would have been smoother if we ported earlier in the RC cycle too and it would be a shame if that happens here. Now is the time to do that.

The dotty team is really responsive to issues. They merged fixes for issues I opened earlier in a matter of hours after reporting. I can check tomorrow what issues are still open and give them a ping and then we can open new tickets for what's still pending.

Then maybe link them here, so it's easy to keep track what workarounds can be undone/what blockers there still are, so you can give that a nudge every now and then if it doesn't move.

Getting scalatest working and getting CI going for this repo will also help. I'm not sure what the plan is for the scalatest people, whether they'll release everything to 0.26 or just the latest version. I see they have a PR open for 0.26 but I don't know how soon they plan on releasing that.

Moving this repo to scalatest 3.2 was kind of annoying, but not too bad, even without the scalafix they have. If you still need to upgrade dependencies company-wide, then there isn't too much use in a PR for that I would think?

[larsrh]

I have a bit time at my hands. Would you be interested in me sending a PR porting to ScalaTest 3.2? (I think this is a good idea regardless of Dotty.)

[martijnhoekstra]

Reflective access is fixed in https://github.com/lampepfl/dotty/pull/9420 -- it's not released yet. To workaround, we could maybe use a scala-2 specific shim. That probably needs some sbt-foo to get a scala-2 only directory analogous to the 2.13+/2.12- directories. Or is that built-in in the dotty plugin?

The thing with scalatest is that (at least it used to be the case that) dependencies have to be updated twitter-wide, so it's broader than just this repo.

[larsrh]

The thing with scalatest is that (at least it used to be the case that) dependencies have to be updated twitter-wide, so it's broader than just this repo.

Yeah, @felixbr told me as much. Based on a quick investigation it also requires

• running the Scalafix rules to avoid the deprecated names in ScalaTest (can be done with ST 3.1)
• upgrading Mockito

[martijnhoekstra]

What I thought was a varargs issue now looks like an overriding bug in dotty: https://github.com/lampepfl/dotty/issues/9463

[mosesn]

I'm talking to a team in a couple hours that might be willing to move Twitter to scalatest 3.2, let me get back to you. If they don't have bandwidth, I can (probably) do it myself.

Re: Config, I think we can delete it, it has been deprecated for 7 years. That'll be an action item for me, we have to make a few changes in our codebase.

[martijnhoekstra]

That leaves the varargs overriding above that I haven't yet found a workaround for, and scalatest 3.2 actually getting released for dotty. The scala2 shim for reflectiveSelectable works fine, at the cost of some sbt config and a scala2-specific source directory.

[felixbr]

I've updated the issue with the action items you talked about and the two external dependencies we're waiting for.

edit: This time for real, the first edit got lost somehow...

[mosesn]

The meeting we had yesterday wasn't fully resolved, I'll take a stab at applying the scalafix patch (it's complicated because of the scale of our codebase) and see how it goes. Unfortunately our scalafix experts are on vacation, but hopefully I don't run into anything too bad 🤞

[martijnhoekstra]

I asked about the dotty 0.29 plans of scalatest on their mailing list, but my message is still in moderation.

[martijnhoekstra]

I see there is a PR open now to bring 0.29 support to scalatest 3.1.x which I seemingly prematurely assumed to be EOL.

[mosesn]

Rad! I'm not going to try to push moving Twitter to 3.2 forward immediately then. I've taken a stab at Config, but it seems like we used it in a bad way with util-eval and we're going to need to unravel that.

[smarter]

I see there is a PR open now to bring 0.29 support to scalatest 3.1.x which I seemingly prematurely assumed to be EOL.

FWIW, it looks like scalatest 3.1 hasn't been published for dotty 0.27.0-RC1, so it might in fact be EOL by now: https://repo1.maven.org/maven2/org/scalatest/scalatest_0.27/, though it might be worth asking the maintainers.

[mosesn]

Ok, that's good to know. We'll start trying to push for scalatest 3.2. Scalafix seems to have a hard time on codebases of our size, so we're going to need to do some work to get us there.

[martijnhoekstra]

Scalafix seems to have a hard time on codebases of our size

Bug reports about that are probably appreciated, even if they don't turn out to be actionable in a timeframe to be useful to this issue.