scala-cli creates a lot of temporary directories that are never cleaned up

[julienrf]

Every time I run a scala-cli script, scala-cli creates a lot of temporary directories (there seems to be at least one directory per library dependency), which are never cleaned up.

Would it be possible to remove those temporary directories after the program execution?

[Gedochao]

Hey, sorry for the late answer.

We are considering to clean some of the build-relevant directories between scala-cli runs, but we've yet to reach a consensus on what exactly can be cleaned and when. Not sure when this will be done.

In the meantime, you can rely on the clean sub-command.

scala-cli clean .

[julienrf]

Why are these directories necessary? If they are like a cache, should they be moved to the .scala-build directory?

[lwronski]

Because these directories contain compiled Scala files and managed files per project, so this is something like a target directory for sbt.

[julienrf]

I would expect the local .scala-build directory to be used for that, no?

[Gedochao]

Can you provide us with the relevant output of tree -a? And point out which directories do you mean? I assumed we were talking about the contents of .scala-build and .bsp

[Gedochao]

I did a quick test with a script containing a single dependency.

//> using lib "com.lihaoyi::os-lib:0.8.1"
println(os.pwd)

and ran it as you described

▶ scala-cli script.sc
Compiling project (Scala 3.1.3, JVM)
Compiled project (Scala 3.1.3, JVM)
/Users/pchabelski/IdeaProjects/scala-cli-tests/scripts-with-deps

my tree output is the following:

▶ tree -a
.
├── .bsp
│   └── scala-cli.json
├── .scala-build
│   ├── .bloop
│   │   ├── project_52eec80df6_52eec80df6-e5cc2ec1d9
│   │   │   ├── bloop-internal-classes
│   │   │   │   └── main-cyCTyRY7SmCNXyYPTTOCJQ==
│   │   │   │       ├── script$.class
│   │   │   │       ├── script.class
│   │   │   │       ├── script.tasty
│   │   │   │       ├── script_sc$.class
│   │   │   │       ├── script_sc.class
│   │   │   │       └── script_sc.tasty
│   │   │   └── project_52eec80df6_52eec80df6-e5cc2ec1d9-analysis.bin
│   │   └── project_52eec80df6_52eec80df6-e5cc2ec1d9.json
│   ├── ide-inputs.json
│   ├── ide-options-v2.json
│   └── project_52eec80df6_52eec80df6-e5cc2ec1d9
│       ├── classes
│       │   └── main
│       │       ├── script$.class
│       │       ├── script.class
│       │       ├── script.tasty
│       │       ├── script_sc$.class
│       │       ├── script_sc.class
│       │       └── script_sc.tasty
│       └── src_generated
│           └── main
│               └── script.scala
└── script.sc

11 directories, 19 files

I can't see anything beyond the .scala-build and .bsp dirs being generated (?) Is there anything I'm missing to replicate this?

[alexarchambault]

Something's not clear in the discussion above: I assume @julienrf means temporary files created under the OS temp dir (/tmp on Linux, /private/var/folders/... on macOS, ...), not those under .scala-build, right?

Scala CLI can indeed create such temporary files, and not delete them, when its process ends by spawning another one (typically scala-cli run, but also scala-cli repl I think, ...). When run as a native executable, scala-cli uses system calls of the execve family, via this. execve starts a new command, by having it take over the current process, just like exec does in shell scripts (taking over the shell process). As a benefit, the Scala CLI process doesn't have to wait for the child process to exit and then exit with the same exit code. As execve makes the current process become the one of the command we launch, the current process will exit with the command exit code. There's only a single process running overall.

A downside of execve stuff is that the JVM system hooks aren't run: the current JVM process abruptly becomes the one of the command we launch, and that's it.

To work around that, we could either replace calls to os.temp by a method of ours, and ensuring ourselves that directories created this way are cleaned up right before execve. (Or maybe we could create temp files somewhere under .scala-build, but we'd need to find a way to clean them up anyway.)

[julienrf]

Thank you for your comment Alexandre, yes my issue is about the hundreds of directories created under the OS temp dir.