sbt-assembly icon indicating copy to clipboard operation
sbt-assembly copied to clipboard

Published 20 hours ago • verified publisher icon sbt

Failure running assembly with ShadeRules inside Windows Subsystem for Linux (Ubuntu)

Open nivekastoreth opened this issue 6 years ago • 1 comments

Steps:

git clone [email protected]:nivekastoreth/sbt-performance-project.git
cd sbt-performance-project
sbt clean assembly

Expected results:

Assembly jar is built.

Actual results:

Two scenarios, both similar.

  1. Shading claims file does not exist, file does exist
  2. Shading fails because the expected file is a directory

Notes:

  • This same build works on the same physical drive / directory structure when run via cygwin or otherwise not in the WSL layer.
  • I've attempted this using both -Dsbt.io.jdktimestamps=false and -Dsbt.io.jdktimestamps=true, neither of which seems to have any real impact.
  • I've attempted this using -Dsbt.version=1.1.0 and -Dsbt.version=1.1.6 with similar results
  • If I re-run assembly repeatedly without cleaning in between, it will eventually complete its run without failures
  • The file that is fails on is not consistent (in either scenario), nor is it a path length issue since other files with longer names exist in the same folder and the same file will succeed in a subsequent run

Scenario No. 1: Shading claims file does not exist, file does exist

jmaki@jmaki-pc /build/wsl/akashi
$ sbt -Dsbt.log.noformat=true clean assembly | tee sbt-assembly.log
[error] java.io.FileNotFoundException: /build/wsl/akashi/akashi-assembler/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$findMapM$2.class (No such file or directory)
[error]         at java.io.FileOutputStream.open0(Native Method)
[error]         at java.io.FileOutputStream.open(FileOutputStream.java:270)
[error]         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
[error]         at sbt.io.Using$.$anonfun$fileOutputStream$1(Using.scala:86)
[error]         at sbt.io.Using$$anon$2.openImpl(Using.scala:76)
[error]         at sbt.io.OpenFile.open(Using.scala:43)
[error]         at sbt.io.OpenFile.open$(Using.scala:39)
[error]         at sbt.io.Using$$anon$2.open(Using.scala:75)
[error]         at sbt.io.Using$$anon$2.open(Using.scala:75)
[error]         at sbt.io.Using.apply(Using.scala:21)
[error]         at sbt.io.IO$.writeBytes(IO.scala:820)
[error]         at sbt.io.IO$.write(IO.scala:817)
[error]         at sbtassembly.Shader$.$anonfun$shadeDirectory$7(Shader.scala:103)
[error]         at sbtassembly.Shader$.$anonfun$shadeDirectory$7$adapted(Shader.scala:97)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:944)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:944)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:71)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error]         at sbtassembly.Shader$.shadeDirectory(Shader.scala:97)
[error]         at sbtassembly.Assembly$.$anonfun$assembleMappings$9(Assembly.scala:218)
[error]         at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:112)
[error]         at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:109)
[error]         at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:62)
[error]         at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1052)
[error]         at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:49)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error]         at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:63)
[error]         at scala.collection.parallel.Task.tryLeaf(Tasks.scala:52)
[error]         at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:46)
[error]         at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1049)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:166)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:153)
[error]         at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:436)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:146)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:145)
[error]         at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:436)
[error]         at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[error] (assembler / assembly / assembledMappings) java.io.FileNotFoundException: /build/wsl/akashi/akashi-assembler/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$findMapM$2.class (No such file or directory)

jmaki@jmaki-pc /build/wsl/akashi
$ grep 'EphemeralStream$$anonfun$findMapM$2.class' sbt-assembly.log -C 2
[debug]         scalaz/EphemeralStream$$anonfun$findMapM$2$$anonfun$apply$10.class
[debug]           /build/wsl/akashi/akashi-preprocessor/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$findMapM$2$$anonfun$apply$10.class
[debug]         scalaz/EphemeralStream$$anonfun$findMapM$2.class
[debug]           /build/wsl/akashi/akashi-preprocessor/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$findMapM$2.class
[debug]         scalaz/EphemeralStream$$anonfun$flatMap$1.class
[debug]           /build/wsl/akashi/akashi-preprocessor/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$flatMap$1.class
--
[debug] Including: macro-compat_2.11-1.1.1.jar
[info] Done packaging.
[error] java.io.FileNotFoundException: /build/wsl/akashi/akashi-assembler/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$findMapM$2.class (No such file or directory)
[error]         at java.io.FileOutputStream.open0(Native Method)
[error]         at java.io.FileOutputStream.open(FileOutputStream.java:270)
--
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[error] (assembler / assembly / assembledMappings) java.io.FileNotFoundException: /build/wsl/akashi/akashi-assembler/target/streams/$global/assemblyOption/$global/streams/assembly/d0da05e6b00b2d3451689652163711f0901137a8_487a90889a8cc17f04a2ab8a290882c255a32c44_dc564712cc91549a72808653b6df35683be7efbf/scalaz/EphemeralStream$$anonfun$findMapM$2.class (No such file or directory)
[error] Total time: 272 s, completed Aug 2, 2018 8:36:56 PM

Scenario No. 2: Shading fails because the expected file is a directory

[error] java.io.FileNotFoundException: /build/wsl/akashi/akashi-assembler/target/streams/$global/assemblyOption/$global/streams/assembly/28139657d8472fc9db06ed0b9d0df2eaefed629f_5fa98cd1a63c99a44dd8d3b77e4762b066a5d0c5_dc564712cc91
549a72808653b6df35683be7efbf/com/google/common/util/concurrent (Is a directory)
[error]         at java.io.FileInputStream.open0(Native Method)
[error]         at java.io.FileInputStream.open(FileInputStream.java:195)
[error]         at java.io.FileInputStream.<init>(FileInputStream.java:138)
[error]         at sbt.io.Using$.$anonfun$fileInputStream$1(Using.scala:88)
[error]         at sbt.io.Using$$anon$2.openImpl(Using.scala:76)
[error]         at sbt.io.OpenFile.open(Using.scala:43)
[error]         at sbt.io.OpenFile.open$(Using.scala:39)
[error]         at sbt.io.Using$$anon$2.open(Using.scala:75)
[error]         at sbt.io.Using$$anon$2.open(Using.scala:75)
[error]         at sbt.io.Using.apply(Using.scala:21)
[error]         at sbt.io.IO$.readBytes(IO.scala:789)
[error]         at sbtassembly.Shader$.$anonfun$shadeDirectory$7(Shader.scala:98)
[error]         at sbtassembly.Shader$.$anonfun$shadeDirectory$7$adapted(Shader.scala:97)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:944)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:944)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:71)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error]         at sbtassembly.Shader$.shadeDirectory(Shader.scala:97)
[error]         at sbtassembly.Assembly$.$anonfun$assembleMappings$9(Assembly.scala:218)
[error]         at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:112)
[error]         at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:109)
[error]         at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:62)
[error]         at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1052)
[error]         at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:49)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error]         at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:63)
[error]         at scala.collection.parallel.Task.tryLeaf(Tasks.scala:52)
[error]         at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:46)
[error]         at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1049)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:156)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:153)
[error]         at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:436)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:146)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:145)
[error]         at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:436)
[error]         at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
[error]         at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
[error]         at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:375)
[error]         at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:375)
[error]         at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:436)
[error]         at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:419)
[error]         at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:412)
[error]         at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56)
[error]         at scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult(Tasks.scala:551)
[error]         at scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult$(Tasks.scala:551)
[error]         at scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:80)
[error]         at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:956)
[error]         at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:49)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error]         at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:63)
[error]         at scala.collection.parallel.Task.tryLeaf(Tasks.scala:52)
[error]         at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:46)
[error]         at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:951)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:149)
[error]         at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:145)
[error]         at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:436)
[error]         at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[error] (assembler / assembly / assembledMappings) java.io.FileNotFoundException: /build/wsl/akashi/akashi-assembler/target/streams/$global/assemblyOption/$global/streams/assembly/28139657d8472fc9db06ed0b9d0df2eaefed629f_5fa98cd1a63c99a44dd8d3b77e4762b066a5d0c5_dc564712cc91549a72808653b6df35683be7efbf/com/google/common/util/concurrent (Is a directory)

Naive Analysis

Both scenarios have the same stacktrace, centered around around:

[error]         at sbt.io.IO$.readBytes(IO.scala:789)
[error]         at sbtassembly.Shader$.$anonfun$shadeDirectory$7(Shader.scala:98)
[error]         at sbtassembly.Shader$.$anonfun$shadeDirectory$7$adapted(Shader.scala:97)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:944)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:944)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:71)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error]         at sbtassembly.Shader$.shadeDirectory(Shader.scala:97)
[error]         at sbtassembly.Assembly$.$anonfun$assembleMappings$9(Assembly.scala:218)

This section of code is of specific interest since line 97 should preclude directory files from being included in the set being read: https://github.com/sbt/sbt-assembly/blob/661fee89d1fb9c84fe25919cb92fc6bb9438b825/src/main/scala/sbtassembly/Shader.scala#L97-L98

This contradicts the error being reported by scenario no. 2, in which IO.readBytes is invoked with a directory path.

This, in some part, is because _.isDirectory returns false for all situations in which the file being checked isn't a directory, including not existing.

    /**
     * Tests whether the file denoted by this abstract pathname is a
     * directory.
     *
     * <p> Where it is required to distinguish an I/O exception from the case
     * that the file is not a directory, or where several attributes of the
     * same file are required at the same time, then the {@link
     * java.nio.file.Files#readAttributes(Path,Class,LinkOption[])
     * Files.readAttributes} method may be used.
     *
     * @return <code>true</code> if and only if the file denoted by this
     *          abstract pathname exists <em>and</em> is a directory;
     *          <code>false</code> otherwise
     *
     * @throws  SecurityException
     *          If a security manager exists and its <code>{@link
     *          java.lang.SecurityManager#checkRead(java.lang.String)}</code>
     *          method denies read access to the file
     */
    public boolean isDirectory()

Which relies on:

    /**
     * Return the simple boolean attributes for the file or directory denoted
     * by the given abstract pathname, or zero if it does not exist or some
     * other I/O error occurs.
     */
    public abstract int getBooleanAttributes(File f);

This all seems to point to some race condition in which a list of files/paths is built by sbtassembly.AssemblyUtils#getMappings while, somehow, those files/paths are still being created, and it's down to a race condition as to whether or not those folders/files will be completely flushed to disk by the time getMappings is called (and/or by the time shadeDirectory starts iterating over those mappings).

But, as the title of this section points out, this is only my naive analysis.

Environment:

WSL Environment:

jmaki@jmaki-pc /build/wsl/akashi
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

jmaki@jmaki-pc /build/wsl/akashi
$ sbt sbtVersion
[info] Loading settings for project akashi-build from plugin-resolvers.sbt,plugins.sbt ...
[info] Loading project definition from /build/wsl/akashi/project
[info] Loading settings for project root from build.sbt,version.sbt ...
[info] Set current project to akashi (in build file:/build/wsl/akashi/)
[info] common / sbtVersion
[info]  1.2.0
[info] preprocessor / sbtVersion
[info]  1.2.0
[info] assembler / sbtVersion
[info]  1.2.0
[info] sbtVersion
[info]  1.2.0

jmaki@jmaki-pc /build/wsl/akashi
$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

Windows Environment

OS Name      Microsoft Windows 10 Pro
Version      10.0.17134 Build 17134
Processor    Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz, 4008 Mhz, 4 Core(s), 8 Logical Processor(s)

nivekastoreth avatar Aug 02 '18 20:08 nivekastoreth

I've updated the description (reproduction steps) to include a link to the repo I've updated to reproduce this issue: https://github.com/nivekastoreth/sbt-performance-project

nivekastoreth avatar Aug 02 '18 21:08 nivekastoreth