Windows - lingering java process locks file from deletion, breaks compilation

[james-s-w-clark]

I'm checking compilation locally to reduce CI job runs (that's a different story though).

When I re-run a compilation (mill contrib.flyway.compile), I get an error - which I can solve by killing the only "java.exe" process. This stackoverflow answer helped me get the compilation running again. Windows wouldn't let me delete the Jar:

mill contrib.flyway.compile 
...
[info] done compiling
[99/365] de.tobiasroeser.mill.vcs.version.VcsVersion.vcsState.overridden.de.tobiasroeser.mill.vcs.version.VcsVersion.vcsState
[177/365] main.moduledefs.manifest
Exception in thread "MillServerActionRunner" java.nio.file.FileSystemException: C:\Users\user\mill\out\main\moduledefs\jar.dest\out.jar: The process cannot access the file because it is being used by another process
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:275)
        at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
        at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
        at os.remove$.apply(FileOps.scala:290)
        at os.remove$.apply(FileOps.scala:284)
        at os.remove$all$.$anonfun$apply$5(FileOps.scala:301)
        at os.remove$all$.$anonfun$apply$5$adapted(FileOps.scala:301)
        at geny.Generator.$anonfun$foreach$1(Generator.scala:50)
        at geny.Generator$Mapped.$anonfun$generate$4(Generator.scala:283)
        at os.walk$stream$$anon$2$$anon$3.visitFile(ListOps.scala:249)
        at os.walk$stream$$anon$2$$anon$3.visitFile(ListOps.scala:230)
        at java.base/java.nio.file.Files.walkFileTree(Files.java:2811)
        at os.walk$stream$$anon$2.generate(ListOps.scala:230)
        at geny.Generator$Mapped.generate(Generator.scala:283)
        at geny.Generator.foreach(Generator.scala:49)
        at geny.Generator.foreach$(Generator.scala:49)
        at geny.Generator$Mapped.foreach(Generator.scala:281)
        at os.remove$all$.apply(FileOps.scala:301)
        at mill.eval.Evaluator.evaluateGroupCached(Evaluator.scala:463)
        at mill.eval.Evaluator.$anonfun$sequentialEvaluate$2(Evaluator.scala:202)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1293)
        at mill.eval.Evaluator.sequentialEvaluate(Evaluator.scala:177)
        at mill.eval.Evaluator.evaluate(Evaluator.scala:162)
        at mill.main.RunScript$.evaluateNamed(RunScript.scala:364)
        at mill.main.RunScript$.evaluate(RunScript.scala:349)
        at mill.main.RunScript$.$anonfun$evaluateTasks$1(RunScript.scala:314)
        at scala.util.Either.map(Either.scala:382)
        at mill.main.RunScript$.evaluateTasks(RunScript.scala:312)
        at mill.main.RunScript$.$anonfun$runScript$8(RunScript.scala:105)
        at ammonite.util.Res$Success.flatMap(Res.scala:62)
        at mill.main.RunScript$.runScript(RunScript.scala:104)
        at mill.main.MainRunner.$anonfun$runScript$1(MainRunner.scala:119)
        at mill.main.MainRunner.watchLoop2(MainRunner.scala:67)
        at mill.main.MainRunner.runScript(MainRunner.scala:92)
        at mill.MillMain$.main0(MillMain.scala:310)
        at mill.main.MillServerMain$.main0(MillServerMain.scala:79)
        at mill.main.Server.$anonfun$handleRun$1(MillServerMain.scala:183)
        at java.base/java.lang.Thread.run(Thread.java:833)

I've worked around it for now. I raised this in case someone else gets the same compilation problems on Windows.

[lefou]

This looks like the jar is either still in use, or some process forget to close the file resource. If you don't have other processes using this jar file, chances are, that Mill itself (or zinc) is keeping it open. I already fixed some resource leaks in Mill in the past, to get Windows CI working, so there is definitely code which was written without resources management in mind. Although I reviewed many sensitive parts, any help is much appreciated.

[james-s-w-clark]

@lefou I'll try and have a look. It seems there's already some Zinc issues for local tests on Windows:

X mill.integration.forked.ZincIncrementalCompilationTests.incremental compilation only compiles changed files 13634ms 
  utest.AssertionError: assert(modelInfo1.ctime == modelInfo2.ctime)
  modelInfo1: os.StatInfo = StatInfo(5641,2022-07-12T08:17:19.8867586Z,2022-07-12T08:17:20.006868Z,2022-07-12T08:17:19.8857575Z,File)
  modelInfo2: os.StatInfo = StatInfo(5641,2022-07-12T08:17:19.8867586Z,2022-07-12T08:17:24.5620146Z,2022-07-12T08:17:19.8857575Z,File)
    utest.asserts.Asserts$.assertImpl(Asserts.scala:30)
    mill.integration.ZincIncrementalCompilationTests.$anonfun$tests$2(ZincIncrementalCompilationTests.scala:39)
1 targets failed
integration.forked.test 1 tests failed:
  mill.integration.forked.ZincIncrementalCompilationTests mill.integration.forked.ZincIncrementalCompilationTests.incremental compilation only compiles changed files

If you could link any issues/PRs/discussion on Zinc to help me understand what it is, it'd be very helpful. I don't know much about how Mill works etc. 👍

[lefou]

@lefou I'll try and have a look. It seems there's already some Zinc issues for local tests on Windows:

X mill.integration.forked.ZincIncrementalCompilationTests.incremental compilation only compiles changed files 13634ms 
  utest.AssertionError: assert(modelInfo1.ctime == modelInfo2.ctime)
  modelInfo1: os.StatInfo = StatInfo(5641,2022-07-12T08:17:19.8867586Z,2022-07-12T08:17:20.006868Z,2022-07-12T08:17:19.8857575Z,File)
  modelInfo2: os.StatInfo = StatInfo(5641,2022-07-12T08:17:19.8867586Z,2022-07-12T08:17:24.5620146Z,2022-07-12T08:17:19.8857575Z,File)
    utest.asserts.Asserts$.assertImpl(Asserts.scala:30)
    mill.integration.ZincIncrementalCompilationTests.$anonfun$tests$2(ZincIncrementalCompilationTests.scala:39)
1 targets failed
integration.forked.test 1 tests failed:
  mill.integration.forked.ZincIncrementalCompilationTests mill.integration.forked.ZincIncrementalCompilationTests.incremental compilation only compiles changed files

If you could link any issues/PRs/discussion on Zinc to help me understand what it is, it'd be very helpful. I don't know much about how Mill works etc. +1

Those are new and probably introduced between Zinc 1.7.0-M2 and 1.7.0 (final). E.g. PR #1845 which ran successfully with 1.7.0-M2 before I updated it to 1.7.0. That's also a good example, why I try to isolate version bumps.

[sake92]

@IdiosApps I tried using https://github.com/jenkinsci/lib-file-leak-detector to find which files remain unclosed.
Found that there are 3 places where it gets stuck/unclosed. You can see attached file for stacktraces.
Couldn't find any place where it would be unclosed in Mill codebase... :/

assembly_file_leaks.txt

[lefou]

@lefou I'll try and have a look. It seems there's already some Zinc issues for local tests on Windows:

X mill.integration.forked.ZincIncrementalCompilationTests.incremental compilation only compiles changed files 13634ms 
  utest.AssertionError: assert(modelInfo1.ctime == modelInfo2.ctime)
  modelInfo1: os.StatInfo = StatInfo(5641,2022-07-12T08:17:19.8867586Z,2022-07-12T08:17:20.006868Z,2022-07-12T08:17:19.8857575Z,File)
  modelInfo2: os.StatInfo = StatInfo(5641,2022-07-12T08:17:19.8867586Z,2022-07-12T08:17:24.5620146Z,2022-07-12T08:17:19.8857575Z,File)
    utest.asserts.Asserts$.assertImpl(Asserts.scala:30)
    mill.integration.ZincIncrementalCompilationTests.$anonfun$tests$2(ZincIncrementalCompilationTests.scala:39)
1 targets failed
integration.forked.test 1 tests failed:
  mill.integration.forked.ZincIncrementalCompilationTests mill.integration.forked.ZincIncrementalCompilationTests.incremental compilation only compiles changed files

If you could link any issues/PRs/discussion on Zinc to help me understand what it is, it'd be very helpful. I don't know much about how Mill works etc. +1

Those are new and probably introduced between Zinc 1.7.0-M2 and 1.7.0 (final). E.g. PR #1845 which ran successfully with 1.7.0-M2 before I updated it to 1.7.0. That's also a good example, why I try to isolate version bumps.

Those are fixed now in Zinc 1.7.1, which was merged into master.

[lefou]

@IdiosApps I tried using https://github.com/jenkinsci/lib-file-leak-detector to find which files remain unclosed. Found that there are 3 places where it gets stuck/unclosed. You can see attached file for stacktraces. Couldn't find any place where it would be unclosed in Mill codebase... :/

assembly_file_leaks.txt

Much appreciated! Yeah, looks like those were opened in Zinc but never closed. I'm not very familiar with sbt/zinc code base, but just from looking at it, I'd bet it's the plugin loading mechanism. Resources issues in normal classpath handling would be probably detected by many others, whereas comsuming locally built plugins which also change frequently is probably rather uncommon.

[lefou]

I reported the issue upstream.

• https://github.com/sbt/zinc/issues/1124