`-Wunused:all` is slow to compile

[oyvindberg]

Compiler version

3.4.0-RC4

Problem

First I want to celebrate the work that has been done for Scala 3.4 to provide Intellij with timing details so that you can see which phases are slow, which files cause much compilation time and so on. Great work!

So at $WORK we just migrated to Scala 3, and are facing slow compiles. Now I finally had the tools to dig a bit deeper. The first thing I found was that the two unused phases consume more time than typer for many of our sbt modules! I'll show an illustrative example here:

The data shown there seems to be correct, and can be reproduced with sbt.

Clean/compile for the same module with -Wunused:all enabled (slightly shorter times because of warmer compiler)

[success] Total time: 15 s, completed Feb 12, 2024, 12:15:43 AM
[success] Total time: 16 s, completed Feb 12, 2024, 12:16:11 AM
[success] Total time: 16 s, completed Feb 12, 2024, 12:16:11 AM

And without:

[success] Total time: 7 s, completed Feb 12, 2024, 12:18:44 AM
[success] Total time: 8 s, completed Feb 12, 2024, 12:18:58 AM
[success] Total time: 7 s, completed Feb 12, 2024, 12:19:11 AM

I'm also very happy that the unused functionality is back, and hope it can be optimized a bit. For now it's very easy to add all the flags from sbt-tpolecat, end up with a slow build and conclude that "Scala is slow to compile".

Reproducing

So this is a private project, but I suspect it will reproduce elsewhere. I tried for instance lichess lila (a prominent public scala 3 repo).

with unused:

[lila] $ user/clean;user/compile
[success] Total time: 0 s, completed Feb 12, 2024, 12:53:50 AM
[info] compiling 28 Scala sources and 1 Java source to /Users/oyvind/pr/lila/modules/user/target/scala-3.3.1/classes ...
[warn] -- Warning: /Users/oyvind/pr/lila/modules/user/src/main/Env.scala:5:35 ---------
[warn] 5 |import lila.common.autoconfig.{ *, given }
[warn]   |                                   ^^^^^
[warn]   |                                   unused import
[warn] -- Warning: /Users/oyvind/pr/lila/modules/user/src/main/Env.scala:10:19 --------
...
[warn] 9 warnings found
[success] Total time: 5 s, completed Feb 12, 2024, 12:53:56 AM
[lila] $ user/clean;user/compile

without unused:

[lila] $ user/clean;user/compile
[success] Total time: 0 s, completed Feb 12, 2024, 12:54:58 AM
[info] compiling 28 Scala sources and 1 Java source to /Users/oyvind/pr/lila/modules/user/target/scala-3.3.1/classes ...
[success] Total time: 3 s, completed Feb 12, 2024, 12:55:01 AM

I can also be of assistance with for instance profiling data if that helps.

Concurrent linting question

I assume compiler trees are immutable and linting phases don't affect the output trees. Did you look into running them in the background as the compiler progresses?

[kyri-petrou]

Just want to say that we've been experiencing a similar issue at $WORK. One of our services uses Scala 3, and the compiling time is roughly ~50-100% longer when -Wunused:linted is used (depending how warm is the compile server). After playing around with the compiler flags a bit, it seems that the majority of the added time comes from -Wunused:imports. Worryingly, version 3.4.0-RCX seems to be noticeably slower when linting is enabled (as opposed to 3.3.x).

I should also point out that since IntelliJ enables Wunused:imports by default for its "Remove unused imports" functionality, large projects end up having very bad dev experience due to the additional compilation time incurred by linting

[hamnis]

In the linked gist this is what I get with a cold compiler for $WORK.

JAVA_OPTS=-Dsbt.task.timings=true sbt clean cronjobs/compile

https://gist.github.com/hamnis/acca674f19cd8cc253ae916f48709fa1

I included the number of source files to indicate size of the module.

[szymon-rd]

Thanks @hamnis for data! It seems that the problem started with Scala 3.4.0, hopefully it is a regression caused by some change that is easy to revert in linting cases.

[dimitarg]

I'm unable to share a repro / something more useful right now, but we have anecdotal reports at work of this being experienced in 3.3.3 as well.

[kyri-petrou]

Hi, in case this might help someone with optimizing this issue, it seems that one good OS repo to replicate this is regression in compiling time is kyo.

Overall, more than half of the compilation time of some modules is spent on checking unused imports:

[matthughes]

@kyri-petrou How did you get that profiling information? Is this something we can print out from the compiler or something internal to Intellij?

[Gedochao]

This seems to be improved considerably by #20321. In case further tweaks are necessary, let's track them under a new ticket. Closing this.