Extract dependencies with the Build Server Protocol (BSP)

[Fristi]

Could scala-steward work with BSP?

This would enable maybe support for Bazel aswell?

[fthomas]

You mean the Build Server Protocol? I've no idea. Scala Steward just needs to get a list of dependencies and resolvers from the build tool. If BSP provides that, I guess it could use BSP to obtain that data.

[fthomas]

Btw, I think it would be super cool if Scala Steward could leverage BSP and work with any BSP implementing build tools out of the box. I imagine that it would also be a fun project to work on.

[fthomas]

I experimented a little bit with sbt and BSP to see if I can extract dependencies via BSP. These were my baby steps:

1. Starting sbt:

$ sbt
...
[info] sbt server started at local:///home/frank/.sbt/1.0/server/1d64d996d87d4bc7b927/sock
...

2. Getting an example JSON-RPC message from https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#contentPart:

{
	"jsonrpc": "2.0",
	"id": 1,
	"method": "textDocument/didOpen",
	"params": {
		...
	}
}

3. Sending a workspace/buildTargets request to the socket mentioned by sbt:

$ echo '{ "jsonrpc": "2.0", "id": 0, "method": "workspace/buildTargets", "params": {} }' | \
  nc -U /home/frank/.sbt/1.0/server/1d64d996d87d4bc7b927/sock

Which worked and sbt responded with a lengthy JSON response that included (among others) a Build Target Identifier { "uri": "file:/home/frank/data/code/scala-steward/core/#core/Compile" }

4. Calling another method with the previous build target id as parameter:

$ echo '{ "jsonrpc": "2.0", "id": 1, "method": "buildTarget/sources", "params": { "targets": [{"uri":"file:/home/frank/data/code/scala-steward/core/#core/Compile"}] } }' | \
  nc -U /home/frank/.sbt/1.0/server/1d64d996d87d4bc7b927/sock

This worked too and sbt responded with a list of source directories for Scala Steward's core project.

5. I then tried calling the buildTarget/dependencyModules where I expected sbt to return infos about dependencies:

$ echo '{ "jsonrpc": "2.0", "id": 1, "method": "buildTarget/dependencyModules", "params": { "targets": [{"uri":"file:/home/frank/data/code/scala-steward/core/#core/Compile"}] } }' | \
  nc -U /home/frank/.sbt/1.0/server/1d64d996d87d4bc7b927/sock

This unfortunately didn't work and I got not response.

And this is where my short journey ends.

[fthomas]

It is not surprising that the buildTarget/dependencyModules call does not return a response. Calling build/initialize

echo '{ "jsonrpc": "2.0", "id": 1, "method": "build/initialize", "params": { "displayName": "test", "version": "0", "bspVersion": "1.0", "rootUri": "file:/home/frank/data/code/scala-steward/core/", "capabilities": { "languageIds": [] } } }' | \
  nc -U /home/frank/.sbt/1.0/server/1d64d996d87d4bc7b927/sock

returns sbt's current BSP capabilities:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "displayName": "sbt",
    "version": "1.7.0-RC2",
    "bspVersion": "2.0.0-M5",
    "capabilities": {
      "compileProvider": ...,
      "testProvider": ...,
      "runProvider": ...,
      "dependencySourcesProvider": true,
      "resourcesProvider": true,
      "canReload": true,
      "jvmRunEnvironmentProvider": true,
      "jvmTestEnvironmentProvider": true
    }
  }
}

and it is missing a "dependencyModulesProvider": true which would indicate that buildTarget/dependencyModules is supported.

[Fristi]

Weird, is it possible to set sbt with debugging on BSP ?

[fthomas]

Maybe. My current understanding is that buildTarget/dependencyModules is just not supported by sbt. But I haven't checked the sbt sources if that is actually the case.

[fthomas]

Ok, checked it. This method is currently not supported: https://github.com/sbt/sbt/blob/v1.7.0-RC2/main/src/main/scala/sbt/internal/server/BuildServerProtocol.scala#L314-L332

[Fristi]

It seems to be supported by bazel-bsp though https://github.com/JetBrains/bazel-bsp/blob/master/server/src/main/java/org/jetbrains/bsp/bazel/server/bsp/BspServerApi.java#L158

[fthomas]

I had a little chat with Mill about https://github.com/scala-steward-org/test-repo-2 via BSP:

$ /home/frank/.cache/mill/download/0.10.5 --bsp --disable-ticker --color false --jobs 1

Content-Length: 240

{ "jsonrpc": "2.0", "id": 1, "method": "build/initialize", "params": { "displayName": "test", "version": "0", "bspVersion": "1.0", "rootUri": "file:/home/frank/data/code/scala-steward/test-repo-2", "capabilities": { "languageIds": [] } } }

Content-Length: 538

{"jsonrpc":"2.0","id":1,"result":{"displayName":"mill-bsp","version":"0.10.5","bspVersion":"2.0.0","capabilities":{"compileProvider":{"languageIds":["java","scala"]},"testProvider":{"languageIds":["java","scala"]},"runProvider":{"languageIds":["java","scala"]},"debugProvider":{"languageIds":[]},"inverseSourcesProvider":true,"dependencySourcesProvider":true,"dependencyModulesProvider":true,"resourcesProvider":true,"buildTargetChangedProvider":false,"jvmRunEnvironmentProvider":true,"jvmTestEnvironmentProvider":true,"canReload":true}}}

Content-Length: 80

{ "jsonrpc": "2.0", "id": 0, "method": "workspace/buildTargets", "params": {} }

Content-Length: 2256

{"jsonrpc":"2.0","id":0,"result":{"targets":[{"id":{"uri":"file:///home/frank/data/code/scala-steward/test-repo-2/hello?id\u003dhello"},"displayName":"hello","baseDirectory":"file:///home/frank/data/code/scala-steward/test-repo-2/hello","tags":["library","application"],"languageIds":["java","scala"],"dependencies":[],"capabilities":{"canCompile":true,"canTest":false,"canRun":true,"canDebug":false},"dataKind":"scala","data":{"scalaOrganization":"org.scala-lang","scalaVersion":"2.13.1","scalaBinaryVersion":"2.13","platform":1,"jars":["file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.1/scala-compiler-2.13.1.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.1/scala-reflect-2.13.1.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.1/scala-library-2.13.1.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/jline/jline/2.14.6/jline-2.14.6.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.12/jansi-1.12.jar"]}},{"id":{"uri":"file:///home/frank/data/code/scala-steward/test-repo-2?id\u003dmill-build"},"displayName":"mill-build","baseDirectory":"file:///home/frank/data/code/scala-steward/test-repo-2","tags":["library","application"],"languageIds":["scala"],"dependencies":[],"capabilities":{"canCompile":false,"canTest":false,"canRun":false,"canDebug":false},"dataKind":"scala","data":{"scalaOrganization":"org.scala-lang","scalaVersion":"2.13.8","scalaBinaryVersion":"2.13","platform":1,"jars":["file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.8/scala-reflect-2.13.8.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline/3.21.0/jline-3.21.0.jar","file:///home/frank/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.9.0/jna-5.9.0.jar"]}}]}}

Content-Length: 186

{ "jsonrpc": "2.0", "id": 1, "method": "buildTarget/dependencyModules", "params": { "targets": [{"uri":"file:///home/frank/data/code/scala-steward/test-repo-2/hello?id\u003dhello"}] } }

Content-Length: 261

{"jsonrpc":"2.0","id":1,"result":{"items":[{"target":{"uri":"file:///home/frank/data/code/scala-steward/test-repo-2/hello?id\u003dhello"},"modules":[{"name":"eu.timepit:refined","version":"0.10.1"},{"name":"org.scala-lang:scala-library","version":"2.13.1"}]}]}}

What's great here is that Mill responded with the library dependencies "modules":[{"name":"eu.timepit:refined","version":"0.10.1"},{"name":"org.scala-lang:scala-library","version":"2.13.1"}] in the response to the buildTarget/dependencyModules call. Unfortunately the resolvers for these dependencies are not included. Scala Steward needs to know where to check for new versions of these dependencies.

[Fristi]

Nice going! I like it that mill already supports this, but I guess also IDE devs would encouter this issue of not being able to resolve these depenencies? I wonder how they do this.

If BSP is fully supported by mill, sbt, bazel, maven, gradle then scala-steward could work for all of these

[fthomas]

My guess is that resolvers are not currently part of BSP because IDEs do not require them and they let the build tools do the resolution. Maybe we should ask the BSP maintainers if adding resolvers (+ credentials) to the protocol would be possible or if it is out of scope.

[Fristi]

Created a proposal https://github.com/build-server-protocol/build-server-protocol/issues/344

[adpi2]

This is a cool project. Another build tool that supports BSP is scala-cli.

[fthomas]

To make things a little bit more concrete let's have a look at the information Scala Steward currently gets from sbt via the stewardDependencies task that Scala Steward injects into the sbt build. The output of that task looks like this: https://github.com/scala-steward-org/scala-steward/blob/c17d36778448a6fde2d38254f5806af6ca927241/modules/core/src/test/scala/org/scalasteward/core/buildtool/sbt/parserTest.scala#L18-L31 Each line in the output is either a dependency or a resolver or garbage that is ignored. What is notable here is that the artifactIds have a name as written in the build file and a name plus Scala binary suffix if it is a declared with %% or %%% The latter is used to find the versions of that dependency from the resolvers. For sbt plugins there are also the sbtVersion and scalaVersion attributes that are passed to Coursier for resolution. To get all dependencies for all Scala versions in the build, Scala Steward calls the aforementioned task as + stewardDependencies.

If we get more or less the same information via BSP, Scala Steward could work with all BSP compliant build tools out of the box or with minimal effort.

[fthomas]

I think we can build a prototype for BSP support using Mill even if we currently don't get the resolvers from the build server. We could just use Maven Central as resolver for all reported dependencies and hope that the protocol will be extended in the future. The workflow in Scala Steward acting as a BSP client would then look like this:

• discover the build tool used in a project

• run the command of the discovered build tool to generate the BSP connection details

  • sbt: $ sbt bspConfig creates the connection details in .bsp/sbt.json

  • Mill: $ mill mill.bsp.BSP/install creates the connection details in .bsp/mill-bsp.json

• use the connection details to start and connect to the BSP server

• call build/initialize to initialize the build

• call workspace/buildTargets to get a list of build targets

• call buildTarget/dependencyModules with all build targets from the previous response as parameters to get all dependencies of the build

• call build/shutdown to prepare shutdown of the build server

• call build/exit to exit the build server

[fthomas]

I have a working prototype that implements the above workflow and talks to Mill at https://github.com/scala-steward-org/scala-steward/compare/wip/bsp.

Extracting dependencies from the buildTarget/dependencyModules currently works for Java dependencies but not for Scala dependencies. This is illustrated by these two dependencies in the response:

modules = ArrayList (
  DependencyModule [
    name = "eu.timepit:refined"
    version = "0.10.1"
    dataKind = null
    data = null
  ],
  DependencyModule [
    name = "ch.epfl.scala:bsp4j"
    version = "2.1.0-M3"
    dataKind = null
    data = null
  ],
  ...

These correspond to this definition in the build: https://github.com/scala-steward-org/test-repo-2/blob/4b70c7459a2d47427c6738e811306a3cdaa4de1b/build.sc#L24-L27

eu.timepit:refined is actually a Scala library and its exact artifactId is refined_2.13. So Mill is only telling us the artifactId as written in the build without the Scala binary suffix but Scala Steward requires the exact artifactId to look up available versions in the appropriate resolvers. bsp4j is a pure Java dependency and has no additional suffix, Scala Steward is able to extract and find its latest version and update it (which it did in https://github.com/scala-steward-org/test-repo-2/pull/176).

There are at least two options how Mill could report the exact artifactId in the above DependencyModule:

1. Change the name field such that includes the exact artifactId and not the name that is used in the build, e.g.: name = "eu.timepit:refined_2.13"
2. Use the optional DependencyModule#data field to include a MavenDependencyModule which has separate fields for the organization and name (artifactId) which could use the exact artifactId.

The BSP is not very specific in these cases which values are expected in these fields. I'll look into Mill and see if we can change the implementation such that exact artifactIds are used.

[fthomas]

Tried this prototype with a simple Bazel project but unfortunately bazelbsp is not yet a dependency modules provider (dependencyModulesProvider = null).

[lefou]

There are at least two options how Mill could report the exact artifactId in the above DependencyModule:

1. Change the `name` field such that includes the exact artifactId and not the name that is used in the build, e.g.: `name = "eu.timepit:refined_2.13"`

2. Use the optional `DependencyModule#data` field to include a [`MavenDependencyModule`](https://build-server-protocol.github.io/docs/extensions/maven) which has separate fields for the organization and name (artifactId) which could use the exact artifactId.

The BSP is not very specific in these cases which values are expected in these fields. I'll look into Mill and see if we can change the implementation such that exact artifactIds are used.

I think option 1. is the most straight forward one.

[lefou]

@fthomas Can you test against Mill 0.11.0-M3 or the latest main branch? In 0.11 we change how we handle transitive dependencies and resolve platform and Scala versions earlier, so there is a good chance, that Mill 0.11 already reports the correct full artifactId.

[fthomas]

Just tested it with 0.11.0-M3:

[info]         DependencyModule [
[info]           name = "eu.timepit:refined_2.13"
[info]           version = "0.10.1"
[info]           dataKind = null
[info]           data = null
[info]         ],
[info]         DependencyModule [
[info]           name = "ch.epfl.scala:bsp4j"
[info]           version = "2.1.0-M4"
[info]           dataKind = null
[info]           data = null
[info]         ],

This is awesome, @lefou! Thank you!

The only thing missing now on the BSP side are the resolvers.

[romanowski]

A shameless plug: we have just released the first stable version of Bazel Steward - a bot similar to Scala Stward but dedicated to Bazel.

The BSP-based approach discussed there should have much wider usage than Bazel (e.g. Bleep) but for now, Bazel Stward may be a good option for Bazel users.

[fthomas]

FYI: BspServerType.scala on the wip/bsp branch now lists BSP servers that are known to Scala Steward and states if they can be used to extract dependencies. Currently only Mill reports dependencies via buildTarget/dependencyModules but I believe that Scala CLI and bleep should be able to do that too once they are updated to the latest Bloop version.