zinc icon indicating copy to clipboard operation
zinc copied to clipboard

Published 20 hours ago • verified publisher icon sbt

ClosedFileSystemException caused by null in ZipFileSystem.ensureOpen in IntelliJ 2024.2 with Azul JDK for project and Temurin JDK for compile server

Open fanf opened this issue 1 year ago • 19 comments

steps

  • download IntelliJ 2024.2 community
  • clone rudder repository: git clone https://github.com/Normation/rudder.git
  • cd to rudder/webapp/sources and run ./rudder/rudder-web/src/main/build.sh to init the js/css env so that to avoid latter compilation errors unrelated to the problem
  • create a new project in IntelliJ in intellij from maven pom located in path_of_clone/rudder/webapp/sources/pom.xml
  • download Temurin 21 (https://adoptium.net/), Azul 21 (https://www.azul.com/downloads/#zulu) and JetBrains JDK (https://github.com/JetBrains/JetBrainsRuntime/releases), unarchive, and add them as a JDK in IntelliJ: Project Structure... > Platform Settings > SDK, then + and select the path to the previous root directories of the SDK ; then in "Project Settings > Project", choose either azul or jbr SDK ; Set the language level to 17
  • open IntelliJ settings and :
    • change the incremental compiler to Zinc in Settings... > Build, Exec, Deploy > Compiler > Scala Compiler top of the screen.
    • force the Scala Compile Server to Temurin SDK in the drop down list in Settings... > Build, Exec, Deploy > Compiler > Scala Compiler > Scala Compile Server then JVM > JDK and change stack size to -Xss8m ;
  • try a compilation: CTRL+F9 : it fails with compilation errors.

If you change the incremental compiler to IDEA it works. It used to work with IntelliJ 2024.1.x. It works if Temurin is chosen.

problem

Note: I started by opening a ticket on JetBrains bug tracker (https://youtrack.jetbrains.com/issue/SCL-22939), but the root cause might be in Zinc thus that ticket too.

The compilation fails on the last project (rudder-web) with two stack traces which seems to indicate that the problem might be in Zinc (see at the end).

I found an other project which had a very similar problem: https://github.com/jobrunr/jobrunr/issues/427 The cause was that the classpath resource manager was not thread safe as show in the correction: https://github.com/jobrunr/jobrunr/commit/248e098345d49eabd85bd267f0183c99ece49847

Exception 1 :

scala: ## Exception when compiling 140 sources to /home/fanf/java/workspaces/rudder-project/rudder/webapp/sources/rudder/rudder-web/target/classes
java.nio.file.ClosedFileSystemException
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
java.base/java.nio.file.Files.readAttributes(Files.java:1853)
java.base/java.nio.file.Files.size(Files.java:2462)
sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
scala.util.Try$.apply(Try.scala:217)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
com.facebook.nailgun.NGSession.run(NGSession.java:198)

Exception 2 :

scala: Compilation failed when compiling to: /home/fanf/java/workspaces/rudder-project/rudder/webapp/sources/rudder/rudder-web/target/classes
  null
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
  jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
  jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
  java.base/java.nio.file.Files.readAttributes(Files.java:1853)
  java.base/java.nio.file.Files.size(Files.java:2462)
  sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
  sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
  sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
  sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
  sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
  sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
  sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
  sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
  java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
  sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
  sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
  sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
  scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
  java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
  scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
  sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
  sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
  scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
  scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
  scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
  sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
  scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
  scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
  scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
  sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
  sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
  sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
  sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
  sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
  sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
  sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
  sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
  sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
  sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
  sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
  sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
  sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
  sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
  sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
  sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
  sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
  scala.util.Try$.apply(Try.scala:217)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
  org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
  org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
  org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
  org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
  org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
  org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
  org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
  org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
  org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
  java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base/java.lang.reflect.Method.invoke(Method.java:580)
  com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
  com.facebook.nailgun.NGSession.run(NGSession.java:198)

fanf avatar Aug 13 '24 16:08 fanf

Since Maven also uses Zinc through scala-maven-plugin`, it might be possible to reproduce the exception by just using Maven in the terminal with the JetBrains JDK as above.

vasilmkd avatar Aug 13 '24 16:08 vasilmkd

And... No. But maven plugin is extremely linear and non concurrent contrary to intellij that tries to parallelize as much as possible (for our joy), so if it's a threadsafety problem, it might be consistant (but it's just a guessing game here).

fanf avatar Aug 13 '24 16:08 fanf

download JetBrains JDK (https://github.com/JetBrains/JetBrainsRuntime/releases)

If this doesn't happen on Zulu or Eclipse Temurin, I don't think we'd work on this issue. If there's some logical error in the concurrency handling, we'd be happy to review pull requests though.

eed3si9n avatar Aug 13 '24 19:08 eed3si9n

@eed3si9n Oh ! I didn't test with azul, and actually I have the same error ! So I retested with Temurin... And now I get the error too. In the meantime, I deleted my whole .idea and started from a fresh one, so perhaps changed with that. Let me try a with a fresh user, fresh clone, fresh jvm install so that the reproduction step and qualification is actually OK.

fanf avatar Aug 14 '24 13:08 fanf

@eed3si9n so, fresh linux user, fresh clone, fresh install of intellij, fresh everything. I can reproduce the problem with Azul zulu JDK with the updated steps in top post, ie with setting the scala compile server to use temurin 21 and the project to use zulu 21. If I set the scala compile server to zulu too, then the error disappear. So it seems that the problem arise when the compiler server JDK is not the same as the project.

I can't test setting JBR as a scala compile server JDK because then I get a NoClassDefFoundError: INIT (perhaps a limitation of JBR).

So I don't know if the problem is in IntelliJ or zinc, but I does not seems to be directly linked to JBR.

fanf avatar Aug 14 '24 16:08 fanf

For any guys investigating the issue but got hit with the following error when running ./rudder/rudder-web/src/main/build.sh

npm error code 1
npm error path /Users/jiahuitan/rudder/webapp/sources/rudder/rudder-web/src/main/node_modules/elm
npm error command failed
npm error command sh -c node install.js
npm error -- ERROR -----------------------------------------------------------------------
npm error
npm error I tried to get `elm` from @elm_binaries/darwin_arm64, but something went wrong.
npm error This can happen if you use the "--omit=optional" or "--no-optional" npm flag, or
npm error if your "node_modules" folder was copied over from a different computer (or VM).

Try first do a npm install elm under the directory rudder/webapp/sources/rudder/rudder-web/src/main/ and that bypasses the issue for me

Friendseeker avatar Sep 15 '24 04:09 Friendseeker

@fanf May I ask if the issue is reproducible on Apple Silicon MacBooks?

I am trying to reproduce the issue but I am stuck with the last step "try a compilation: CTRL+F9 : it fails with compilation errors.". When I click Build -> Build Project, nothing happens on my M1 Mac.

Update: I resolved the issue of Build -> Build Project having no response. I just disabled all non essential downloaded IDE plugins.

However on my M1 Mac, "Build Project" succeeded.... I guess a Linux machine is needed to debug the issue then.

Update 2:

I clicked "Build Project" for several times and I reproduced the same stack trace! I shall start my investigation. So apparently the issue is flaky... Further suggesting that the thread safety hypothesis might be true.

scala: ## Exception when compiling 142 sources to /Users/jiahuitan/rudder/webapp/sources/rudder/rudder-web/target/classes
java.nio.file.ClosedFileSystemException
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
java.base/java.nio.file.Files.readAttributes(Files.java:1853)
java.base/java.nio.file.Files.size(Files.java:2462)
sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
scala.util.Try$.apply(Try.scala:217)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
com.facebook.nailgun.NGSession.run(NGSession.java:198)

Friendseeker avatar Sep 15 '24 23:09 Friendseeker

@vasilmkd May I ask if it is possible to access the source code of org.jetbrains.jps.incremental.scala.local.SbtCompiler somewhere?

Here's my interim hypothesis of the issue. SbtCompiler called sbt.internal.inc.IncrementalCompilerImpl.compile in multiple different threads, and in each of these calls, SbtCompiler supplied the same ReadStamps, and SbtCompiler used a non thread-safe implementation of ReadStamps, causing the issue.

Never mind, SbtCompiler actually passed in a thread safe implementation of ReadStamps.... Guess we need to formulate hypothesis 2. Still feel like this is a thread safety issue though.

Friendseeker avatar Sep 16 '24 01:09 Friendseeker

@Friendseeker : the keyboard shortcut should be just an alias for "build project", so perhaps the problem somehow depends on some local properties. For reference, even if it likely won't help, I use a Lenovo t14s amd gen 3 with a Ryzen 7 PRO 6850U a'd I use Archlinux. I will try with the menu "build project" to check it leads to the same issue. Unfortunately I don't have a Mac to test.

fanf avatar Sep 16 '24 05:09 fanf

It really feels like a threadsafety issue : the fact that maven (which is seldom parallelized) works, the very similar bug /stack trace linked in the top post that says it was becausethe classpath resource manager wasn't threadsafe, the flakiness, the fact that it used to work with previous intellij version...

Also, it only happens in the "rudder-web" subproject, which has several unique properties :

  • it's the last one in the dependency graph,
  • it's the only "webapp", other are internal libs so it has many more resources files, and a war structure.

fanf avatar Sep 16 '24 06:09 fanf

Hi 👋🏻

In Settings > Build, Execution, Deployment > Compiler > Scala Compiler > Scala Compile Server you can disable parallelism during compilation.

I assume this will make the exception go away?

vasilmkd avatar Sep 16 '24 06:09 vasilmkd

Computers are complicated.

So, I tried what @vasilmkd suggested (unchecked the  compile independant modules in parallel, up to 6 threads), and it did have an effect :tada:

But the effect is that it fails earlier. But not in the first or second module either : image

The stacktraces look identical. So it might be something with resources being closed too early.
I don't see what "rudder-template" has that is special, but the order of modules is not what I would have expected (because my human brain would prefer to do "all ldap-inventory sub-tree" before starting one in "rudder" sub-tree). It's a jar, it doesn't even have lots of resources.

fanf avatar Sep 16 '24 07:09 fanf

I mean, as long as all of the dependencies for rudder-templates have been met, it shouldn't matter what exact algorithm the IDEA build system uses (we don't control this in the Scala plugin, we just get called to compile a module).

Can you please confirm that's the case in the screenshot above?

vasilmkd avatar Sep 16 '24 07:09 vasilmkd

Yes totally, the dependency path is OK. And it looks like it fails when it switch to the test part of the project.

fanf avatar Sep 16 '24 07:09 fanf

@fanf I messaged you on Discord, let's please discuss some details there. Thank you!

vasilmkd avatar Sep 16 '24 08:09 vasilmkd

Dropping these 3 links just in case, I haven't been able to make any conclusions yet.

  • https://github.com/jetty/jetty.project/issues/11548#issuecomment-2013296143
  • https://github.com/jetty/jetty.project/pull/11549
  • https://bugs.openjdk.org/browse/JDK-8291712

vasilmkd avatar Sep 16 '24 08:09 vasilmkd

Debugging the issue, the "zip" file in question which throws the ClosedFileSystemException happens to be any .sig file located in the JDK modules (JDK 9+).

This is a really strange issue. I honestly have no clue what we might be doing in IDEA that could be causing this. Any ideas are greatly appreciated.

vasilmkd avatar Sep 16 '24 11:09 vasilmkd

But these .sig files are only hashed by Zinc when there's a mismatch between the Project JDK and the JDK used to run the Zinc compiler.

Which matches

If I set the scala compile server to zulu too, then the error disappear. So it seems that the problem arise when the compiler server JDK is not the same as the project.

https://github.com/sbt/zinc/issues/1383#issuecomment-2289226896

vasilmkd avatar Sep 16 '24 12:09 vasilmkd

When the JDK running the Zinc compiler differs from the Project JDK, then Zinc is instantiated with an instance of sbt.internal.inc.javac.ForkedJavaCompiler.

vasilmkd avatar Sep 16 '24 12:09 vasilmkd