Add bazel integration

[jvican]

I've been working in a bloop worker process in @andyscott's https://github.com/andyscott/rules_scala_annex, which will eventually be merged in https://github.com/bazelbuild/rules_scala in some way or another.

The integration is basic for now, but the most difficult things have already been figured out. The ongoing implementation can be found in https://github.com/jvican/rules_scala_annex/tree/topic/bloop.

Getting a bloop integration that compiles the bazel targets is the first step. Then we can easily extend it to console, test and run. The long-term idea is to make the bloop integration a little bit smarter:

• If there's an active bloop server, act as a client and relay the actions to the active server (which is supposed to have hot compilers). This will prove more important when the server starts to get used by tools like Intellij.
• If there's no active bloop server, start up a nailgun server that can listen to external actions (IDE invocations, CLI requests via the bloop client), and then simulate Bazel actions in process as if it was another client more.

I'm excited to see how this shapes up. More news soon.

UPDATE: We're now looking into how we could implement a complete Bloop integration in the official Bazel rules_scala repository.

[andyscott]

I'm also very excited to see how this shapes up!

[aSapien]

Hi, has there been any progress with this? I'm looking for a way to use scala metals with a Bazel build.

[jvican]

@aSapien I left some thoughts on how this can be implemented in this blog post https://jvican.github.io/posts/integrate-bloop-with-bazel-and-pants.html

I'm currently chatting with @natansil and other peers at Stripe/Wix about potential implementation rules. My thoughts are just an intuition of how such an implementation would work but my understanding of how Bazel works is not deep so there are a few things to improve.

I'd love to have support for Bazel. Any help (commenting here on the contents of the blog post, helping out with an implementation, etc) is very much appreciated.

[aSapien]

@jvican sounds promising, thanks! Will sync with @natansil cc @hmemcpy

[hmemcpy]

Cheers! Let's hope this happens, the discussions with Pants folks sound promising as well! It will also enable much better IntelliJ integration, they already implemented BSP.

[andyscott]

I went ahead and implemented preliminary Bloop support for Bazel. Right now this works for a simple Java project, and it's stubbed out to support Scala.

https://github.com/andyscott/rules_bloop

There's no documentation... yet. Adventurous Bazel + Bloop savvy folks should be able to get it working looking at the Makefile.

[ioleo]

Any updates on this? What is missing / needs improvement? How can I help to make this happen?

[natansil]

I think that there is a key difference between the way bloop optimises builds (caching and pipelining of "large" code modules) and the way "standard, canonical" bazel does it (caching and parallelising of many "small" code units). I think this is the main reason for delay.

[olafurpg]

FWIW, Metals has an experimental Bloop integration with Pants that is implemented as a standalone command-line tool that generates .bloop/*.json files given a Pants build. The interface is like this

pants-bloop $PROJECT_NAME:: $PROJECT2_NAME:: ...

It's possible to export all targets with pants-bloop :: but it's slow for very large builds.

If there is something equivalent to .pants export-classpath export in Pants then you should be able to do something similarly for Bazel.

[olafurpg]

Bloop BTW performs nicely for builds that have a large number of small targets. The latest 1.4.0-RC1 release includes several fixes that improve the performance in esp. larger build graphs (~several thousand targets)

[natansil]

@olafurpg sounds interesting. Bazel analyses the build graph and sends each target separate build request to the build tool/server. So now does bloop works well in the limited scope of a test target?

[olafurpg]

So now does bloop works well in the limited scope of a test target?

Can you elaborate?

[natansil]

Sorry, I meant does bloop works well in the limited scope of a bazel target (when dependent targets have already been compiled)?

[olafurpg]

No-op compiles in Bloop are pretty fast in my experience even with multi-GB classpaths with ~2k jars and >1k targets. The v1.4.0-RC1 release fixed a few issues in this particular area and I think there are still a few opportunities to speed up things (see https://github.com/scalacenter/bloop/issues/1093). In the worst case, I have seen Bloop take 2-3 seconds to do a no-op compiles.

[natansil]

Thank you @olafurpg for the info! How long will cold scala compiler take for a no-op compilation?

[olafurpg]

The scala compiler is not incremental so it doesn’t have no-op compilation. In Bazel iiuc a no-op compilation would be a cache hit.

[psilospore]

Just submitted a draft PR here: https://github.com/bazelbuild/rules_scala/pull/997