Ability to avoid downloading or checking dependencies that are already present locally

[ghostdogpr]

Is your feature request related to a problem? Please describe. I have a script with a number of dependencies, and every time I run the script, it spends some time downloading or checking dependencies.

Downloading https://jitpack.io/org/scala-lang/scala3-library_3/
Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Failed to check https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
...

It can take a long time (more than 1min in CI), but it's useless because all dependencies are already there in the local cache.

Describe the solution you'd like I would like an option that prevents downloading or checking dependencies that are already in the local cache.

Describe alternatives you've considered I found --suppress-outdated-dependency-warning and scala-cli config actions true but they only disable the warnings, they don't stop this process of checking all dependencies. There is also --offline but it would fail in the case where I really don't have the dependency so it's not ideal.

[Gedochao]

Actually, --suppress-outdated-dependency-warning should prevent Scala CLI from checking if your project's dependencies are at their latest, at least as far as a quick look at the code got me. @ghostdogpr did you test it with your script? is the slowdown present?

[ghostdogpr]

Yeah, it removed the warnings but was still triggering all these Downloading/Checking logs 🤔 Only --offline prevented them.

[Gedochao]

Are you verifying on a recent Scala CLI version? These logs are coming from coursier. We bumped it recently in v1.9.1 (newest Scala CLI as of writing this comment is v1.10.0) Can you confirm for me that this occurs to you with v1.9.1 or v1.10.0? I am also not 100% sure the logs aren't sometimes printed by coursier when the re-download does not actually occur, which is why I also asked whether the slowdown occurs when --suppress-outdated-dependency-warning is passed.

[ghostdogpr]

We are using Scala CLI 1.10.0.

I just tested --suppress-outdated-dependency-warning again and the full log looks like this: https://gist.github.com/ghostdogpr/663cdf2e3c2c8004f94e42f5af67d7bc

Note that the dependencies are provided by the CI cache which is copied from ~/.cache/coursier/v1 previous runs. I wonder if something is missing there?

[Gedochao]

I wonder myself. Need to investigate, I suppose.

[ghostdogpr]

@Gedochao some interesting new findings, if we add ~/.cache/scalacli to what our CI cache is storing, it stops downloading most dependencies with --suppress-outdated-dependency-warning, only these but it's very fast:

Wed, 12 Nov 2025 10:19:38 GMT Downloading https://jitpack.io/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:38 GMT Failed to download https://jitpack.io/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:38 GMT Downloading https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:38 GMT Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:38 GMT Checked https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:38 GMT Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:38 GMT Checked https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:38 GMT Failed to download https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:38 GMT Downloading https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml.sha1
Wed, 12 Nov 2025 10:19:38 GMT Failed to download https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml.sha1
Wed, 12 Nov 2025 10:19:38 GMT Starting compilation server
Wed, 12 Nov 2025 10:19:39 GMT Downloading https://jitpack.io/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:39 GMT Failed to download https://jitpack.io/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:39 GMT Downloading https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:39 GMT Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:39 GMT Checked https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/
Wed, 12 Nov 2025 10:19:39 GMT Failed to download https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:39 GMT Downloading https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml.sha1
Wed, 12 Nov 2025 10:19:39 GMT Checking https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:39 GMT Checked https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/maven-metadata.xml
Wed, 12 Nov 2025 10:19:39 GMT Failed to download https://jitpack.io/org/scala-lang/scala3-library_3/maven-metadata.xml.sha1

There is one downside though, when we restore that folder and run Scala CLI's install-home, it doesn't add scala-cli to $HOME/.profile so we have to do it manually in our script.

[lbialy]

do you use snapshots in this script?

[ghostdogpr]

do you use snapshots in this script?

Nope. I can share the whole directives:

//> using scala 3.6.4
//> using options -language:strictEquality -Wsafe-init -experimental -language:experimental.namedTuples -source:future -Xmax-inlines 64 -Wunused:imports
//> using javaOptions --enable-native-access=ALL-UNNAMED --sun-misc-unsafe-memory-access=allow -XX:+IgnoreUnrecognizedVMOptions -XX:+UnlockExperimentalVMOptions -XX:+UseCompactObjectHeaders
//> using repository jitpack
//> using dep com.github.scopt::scopt::4.1.0
//> using dep com.softwaremill.sttp.client3::core:3.11.0
//> using dep com.lihaoyi::os-lib:0.11.5
//> using dep com.lihaoyi::upickle:4.3.2
//> using dep com.lihaoyi::ujson-circe:4.3.2
//> using dep com.github.nscala-time::nscala-time:3.0.0
//> using dep com.softwaremill.magnolia1_3::magnolia:1.3.18
//> using dep com.lihaoyi::sourcecode:0.4.4
//> using dep org.typelevel::cats-core:2.13.0
//> using dep dev.zio::zio:2.1.21
//> using dep dev.zio::zio-interop-cats:23.1.0.5
//> using dep dev.zio::zio-process:0.7.2
//> using dep com.vladkopanev::zio-saga-core:0.6.0
//> using dep dev.optics::monocle-core:3.3.0
//> using dep dev.optics::monocle-macro:3.3.0
//> using dep org.jline:jline:3.30.6
//> using dep org.typelevel::shapeless3-deriving:3.5.0
//> using dep io.circe::circe-generic:0.14.14
//> using dep io.circe::circe-yaml:1.15.0
//> using dep com.lihaoyi::pprint:0.9.3
//> using dep org.snakeyaml:snakeyaml-engine:2.10
//> using dep com.github.kotlin-inquirer:kotlin-inquirer:0.1.0
//> using objectWrapper

[lbialy]

@Gedochao it's caused by //> using repository jitpack directive, I just put these directives into a file with single @main def main = println("done") and once I comment out jitpack (and kotlin-inquirer that needs it), it starts almost immediately without the resolution phase, it's still slower than it should be but that's because it still computes all these hints:

[hint] ./test.scala:7:15
[hint] "os-lib is outdated, update to 0.11.6"
[hint]      os-lib 0.11.5 -> com.lihaoyi::os-lib:0.11.6
[hint] //> using dep com.lihaoyi::os-lib:0.11.5
[hint]               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[hint] ./test.scala:8:15
[hint] "upickle is outdated, update to 4.4.1"
[hint]      upickle 4.3.2 -> com.lihaoyi::upickle:4.4.1
[hint] //> using dep com.lihaoyi::upickle:4.3.2
[hint]               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[hint] ./test.scala:9:15
[hint] "ujson-circe is outdated, update to 4.4.1"
[hint]      ujson-circe 4.3.2 -> com.lihaoyi::ujson-circe:4.4.1
[hint] //> using dep com.lihaoyi::ujson-circe:4.3.2
[hint]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[hint] ./test.scala:14:15
[hint] "zio is outdated, update to 2.1.22"
[hint]      zio 2.1.21 -> dev.zio::zio:2.1.22
[hint] //> using dep dev.zio::zio:2.1.21
[hint]               ^^^^^^^^^^^^^^^^^^^
[hint] ./test.scala:22:15
[hint] "circe-generic is outdated, update to 0.14.15"
[hint]      circe-generic 0.14.14 -> io.circe::circe-generic:0.14.15
[hint] //> using dep io.circe::circe-generic:0.14.14
[hint]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[hint] ./test.scala:24:15
[hint] "pprint is outdated, update to 0.9.4"
[hint]      pprint 0.9.3 -> com.lihaoyi::pprint:0.9.4
[hint] //> using dep com.lihaoyi::pprint:0.9.3
[hint]               ^^^^^^^^^^^^^^^^^^^^^^^^^
[hint] ./test.scala:25:15
[hint] "snakeyaml-engine is outdated, update to 3.0"
[hint]      snakeyaml-engine 2.10 -> org.snakeyaml:snakeyaml-engine:3.0
[hint] //> using dep org.snakeyaml:snakeyaml-engine:2.10
[hint]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We had the discussion to have the option to suppress this "newer version of a dep available" logic in an another issue already.

[lbialy]

With jitpack I get this for like 2s each time:

λ scala-cli run test.scala
https://jitpack.io/org/scala-lang/scala3-library_3/
    0.0% [          ] 0B (0B / s)

[lbialy]

Effectively it seems to behave exactly like it behaves for snapshots here 🫨

[Gedochao]

Ah, I see. Custom repositories are treated differently. Yeah, that's likely the right track here.