Custom repository for Scala artifacts not propagated to bloop

[WojciechMazur]

Version(s) 1.2.0

Describe the bug In the Open Community Build we're building and publishing the Scala 3 compiler artefacts to the custom repository. These artifacts are used to check if projects can be build against given versions of the compiler. It's invoked like:

scala-cli compile . --scala-version=3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT --repository=https://custom/maven/repo/3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT

When compiling with --server=false it works correctly, however, when running with bloop we get exception:

Compiling project (Scala 3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT, JVM (11))
Error compiling project (Scala 3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT, JVM (11))
Error: Unexpected error when compiling repo_d49e65e4dd_6e665ca4ab: sbt.internal.inc.InvalidComponent: Couldn't retrieve module org.scala-lang:scala3-sbt-bridge:3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT:compile
        at sbt.internal.inc.BloopComponentCompiler.$anonfun$compileAndInstall$2(BloopComponentCompiler.scala:276)
        at sbt.internal.inc.BloopComponentCompiler.$anonfun$compileAndInstall$2$adapted(BloopComponentCompiler.scala:257)
        at sbt.io.IO$.withTemporaryDirectory(IO.scala:488)
        at sbt.io.IO$.withTemporaryDirectory(IO.scala:498)
        at sbt.internal.inc.BloopComponentCompiler.$anonfun$compileAndInstall$1(BloopComponentCompiler.scala:257)
        at sbt.internal.inc.BloopComponentCompiler.$anonfun$compileAndInstall$1$adapted(BloopComponentCompiler.scala:255)
        at sbt.io.IO$.withTemporaryDirectory(IO.scala:488)
        at sbt.io.IO$.withTemporaryDirectory(IO.scala:498)
        at sbt.internal.inc.BloopComponentCompiler.compileAndInstall(BloopComponentCompiler.scala:255)
        at sbt.internal.inc.BloopComponentCompiler.$anonfun$compiledBridgeJar$1(BloopComponentCompiler.scala:245)
        at sbt.internal.inc.IfMissing$Define.run(IfMissing.scala:19)
        at sbt.internal.inc.BloopComponentManager.createAndCache$1(BloopComponentManager.scala:46)
        at sbt.internal.inc.BloopComponentManager.$anonfun$files$3(BloopComponentManager.scala:60)
        at sbt.internal.inc.BloopComponentManager.getOrElse$1(BloopComponentManager.scala:38)
        at sbt.internal.inc.BloopComponentManager.$anonfun$files$2(BloopComponentManager.scala:60)
        at sbt.internal.inc.BloopComponentManager$$anon$2.call(BloopComponentManager.scala:89)
        at bloop.ComponentLock.apply(BloopComponentsLock.scala:10)
        at bloop.ComponentLock.apply$(BloopComponentsLock.scala:9)
        at bloop.BloopComponentsLock$.apply(BloopComponentsLock.scala:14)
        at sbt.internal.inc.BloopComponentManager.lock(BloopComponentManager.scala:89)
        at sbt.internal.inc.BloopComponentManager.$anonfun$lockSecondaryCache$1(BloopComponentManager.scala:84)
        at scala.Option.map(Option.scala:230)
        at sbt.internal.inc.BloopComponentManager.lockSecondaryCache(BloopComponentManager.scala:84)
        at sbt.internal.inc.BloopComponentManager.fromSecondary$1(BloopComponentManager.scala:58)
        at sbt.internal.inc.BloopComponentManager.$anonfun$files$6(BloopComponentManager.scala:64)
        at sbt.internal.inc.BloopComponentManager.getOrElse$1(BloopComponentManager.scala:38)
        at sbt.internal.inc.BloopComponentManager.$anonfun$files$5(BloopComponentManager.scala:64)
        at sbt.internal.inc.BloopComponentManager$$anon$2.call(BloopComponentManager.scala:89)
        at bloop.ComponentLock.apply(BloopComponentsLock.scala:10)
        at bloop.ComponentLock.apply$(BloopComponentsLock.scala:9)
        at bloop.BloopComponentsLock$.apply(BloopComponentsLock.scala:14)
        at sbt.internal.inc.BloopComponentManager.lock(BloopComponentManager.scala:89)
        at sbt.internal.inc.BloopComponentManager.lockLocalCache(BloopComponentManager.scala:82)
        at sbt.internal.inc.BloopComponentManager.files(BloopComponentManager.scala:64)
        at sbt.internal.inc.BloopComponentManager.file(BloopComponentManager.scala:72)
        at sbt.internal.inc.BloopComponentCompiler.compiledBridgeJar(BloopComponentCompiler.scala:245)
        at sbt.internal.inc.BloopComponentCompiler$BloopCompilerBridgeProvider.compiledBridge(BloopComponentCompiler.scala:123)
        at sbt.internal.inc.BloopComponentCompiler$BloopCompilerBridgeProvider.fetchCompiledBridge(BloopComponentCompiler.scala:129)
        at sbt.internal.inc.AnalyzingCompiler.getDualLoader(AnalyzingCompiler.scala:354)
        at sbt.internal.inc.AnalyzingCompiler.getCompilerLoader(AnalyzingCompiler.scala:343)
        at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:87)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.compileSources$1(BloopHighLevelCompiler.scala:133)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$7(BloopHighLevelCompiler.scala:155)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$1(BloopHighLevelCompiler.scala:71)
        at bloop.tracing.NoopTracer$.trace(BraveTracer.scala:53)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.timed$1(BloopHighLevelCompiler.scala:70)
        at sbt.internal.inc.bloop.internal.BloopHighLevelCompiler.$anonfun$compile$6(BloopHighLevelCompiler.scala:155)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81)
        at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
        at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
        at monix.eval.internal.TaskExecuteOn$AsyncRegister$$anon$1.run(TaskExecuteOn.scala:71)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: coursierapi.error.SimpleResolutionError$1: Error downloading org.scala-lang:scala3-sbt-bridge:3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT
  not found: /Users/wmazur/.ivy2/local/org.scala-lang/scala3-sbt-bridge/3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/org/scala-lang/scala3-sbt-bridge/3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT/scala3-sbt-bridge-3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT.pom
  not found: https://scala-ci.typesafe.com/artifactory/scala-integration/org/scala-lang/scala3-sbt-bridge/3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT/scala3-sbt-bridge-3.4.2-RC1-bin-8b1618573bdf0c9cd9704ed4772fcec9c213ca32-SNAPSHOT.pom
        at coursierapi.error.SimpleResolutionError.of(SimpleResolutionError.java:11)
        at coursierapi.shaded.coursier.internal.api.ApiHelper$.simpleResError(ApiHelper.scala:336)
        at coursierapi.shaded.coursier.internal.api.ApiHelper$.doFetch(ApiHelper.scala:380)
        at coursierapi.shaded.coursier.internal.api.ApiHelper.doFetch(ApiHelper.scala)
        at coursierapi.Fetch.fetchResult(Fetch.java:244)
        at coursierapi.Fetch.fetch(Fetch.java:239)
        at bloop.DependencyResolution$.resolveDependenciesWithErrors(DependencyResolution.scala:90)
        at bloop.DependencyResolution$.resolveWithErrors(DependencyResolution.scala:65)
        at sbt.internal.inc.BloopComponentCompiler.$anonfun$compileAndInstall$2(BloopComponentCompiler.scala:267)
        ... 58 more

Then investigating the logs you can see that custom repository https://custom/maven/repo/... was not included in the search of artifacts in the bloop

To Reproduce Publish Scala 3 artifacts to custom remote repository, then try to use them in scala-cli

Expected behaviour The custom repositories passed to the scala-cli should allow to find custom Scala 3 artefacts repositories.

[TomasMikula]

I'm also encountering this.

In the meantime, is it possible to set --server=false in some project settings rather than repeat every time on the command line?

[tgodzik]

@TomasMikula is this the same issue with the sbt bridge or something else entirely? Bloop downloaded sbt bridges for a specific Scala version on demand and I think it doesn't have COURSIER_REPOSITORIES env variable set.

@WojciechMazur Did you try setting the variable for the entire VM?

Alternatively, to improve the situation we could set the env variable automatically when starting Bloop and if repositories were set.

[WojciechMazur]

@tgodzik I wasn't aware of this env variable, setting it up seams to fix the problem, thank you.

It might be a good usability feature if we wouldn't need to set env var up, especially since we need to also add the standard repositories as well, but I guess it requires a sever changes to Bloop.

[tgodzik]

It's only an issue if you use a custom compiler version and it's a bit tricky to fix it otherwise. Probably, the easiest way to do it is to provide those env variables when starting bloop with non default repositories.

[TomasMikula]

Ah, upon inspecting the stack trace, it's a different problem:

Error fetching artifacts:
https://my.company.com/repo/ch/epfl/scala/com-microsoft-java-debug-core/0.34.0+20/com-microsoft-java-debug-core-0.34.0+20.jar: not found: https://my.company.com/repo/ch/epfl/scala/com-microsoft-java-debug-core/0.34.0+20/com-microsoft-java-debug-core-0.34.0+20.jar
https://my.company.com/repo/ch/epfl/scala/directory-watcher/0.8.0+6-f651bd93/directory-watcher-0.8.0+6-f651bd93.jar: not found: https://my.company.com/repo/ch/epfl/scala/directory-watcher/0.8.0+6-f651bd93/directory-watcher-0.8.0+6-f651bd93.jar

Usually, artifacts are blocked if there's a known vulnerability. I will try to find out whether that's the case here.

Sorry for the noise.