Better caching for Scala 3 (via ijar or tasty inspection)

[susliko]

trafficstars

Bazel includes the ijar tool, which strips regular JAR files down to their interfaces. Target hashes are computed based on the resulting ijars, ensuring that only the necessary targets are rebuilt when changes do not affect public interfaces (e.g., modifications to implementations or private method signatures).

However, this approach often falls short in practice. For Scala 2, as noted in this issue, scalac adds a ScalaSignature annotation to classes, causing cache misses and unnecessary rebuilds.

For Scala 3, ijar generation is explicitly disabled.

Is the situation with ijars similar for Scala 3? If ijars are not a viable solution, would analyzing TASTy files for interface-level changes be a feasible alternative?

[jjudd]

I agree that it would be great to improve ijar support for Scala or to use some other mechanism to improve detection of interface vs private implementation changes.

We have a discussion started in #1658 that includes a proposal for ijar (as well as many other things). Our current thoughts are to improve the ijar tool to be more Scala aware. If the Bazel folks are not interested in that, then we could explore other alternatives - perhaps there is some way to leverage the Zinc analysis file to accomplish this? Haven't thought much through that.

Part of the proposal covering ijar: https://docs.google.com/document/d/12BmUPpJpworA9ep2P_J00TxXwFRBZDQttUCsoYo0Yao/edit?tab=t.0#heading=h.z65373pwhv7g

[jadenPete]

Is the situation with ijars similar for Scala 3?

We've enabled ijar for Scala 3 targets in lucidsoftware/rules_scala haven't encountered any issues, so I think enabling it for Scala 3 targets should be fine. We have received the following warning:

ijar: skipping unknown attribute: "TASTY".

but I believe this should no longer be an issue, now that the TASTY attribute is explicitly included in ijar.

From what I understand, though, TASTy markes a major shift in how type information is stored between compiler invocations from Scala 2.13's "Pickle" format in that it includes a completely typed representation of the program's AST. I'm of the opinion that this defeats the purpose of using ijar, since most changes to a program's source code will now induce changes in its TASTy files, thus changing the generated ijar and invalidating reverse dependencies' cached compilation actions.

When you consider that and the fact that "scalac doesn’t have to read their .class files anymore; it can read their .tasty files", it may make more sense to develop an alternative to ijar specially for Scala 3 targets, as opposed to modifying ijar. That way, we can strip out information from TASTy files not needed for compilation (e.g. positional information, documentation, private members, etc.), similar to what @susliko described.

[susliko]

I'm of the opinion that this defeats the purpose of using ijar, since most changes to a program's source code will now induce changes in its TASTy files, thus changing the generated ijar and invalidating reverse dependencies' cached compilation actions.

Honestly, I don't think I've ever seen ijar work with Scala

[jadenPete]

Honestly, I don't think I've ever seen ijar work with Scala

Huh, that's odd; it's worked fine in Scala 2 for me. If you add a comment or some whitespace to a Scala file, or change the implementation of a function, do you find that ijar prevents recompilation?

[jadenPete]

When you consider that and the fact that "scalac doesn’t have to read their .class files anymore; it can read their .tasty files", it may make more sense to develop an alternative to ijar specially for Scala 3 targets, as opposed to modifying ijar.

Just an FYI for those following this issue: I'm going to try implementing this. I'll report back on my progress.