[MSHADE-329] Concurrent writes of dependency-reduced-pom.xml adds same exclusion ad infinitum

[jira-importer]

Håkon Hallingstad opened MSHADE-329 and commented

I have a multi-threaded mvn install that seems to halt but ends up using 200% CPU, in about 50% of the invocations. The mvn command used is:

mvn -T1C -nsu -Dmaven.source.skip -Dmaven.javadoc.skip -Dmaven.test.skip install -rf :MODULE

Using mvnDebug I have found out that there are 2 running Java threads, each writing dependency-reduced-pom.xml in two different modules A and B, respectively. These files seems to become several MB large, before they're deleted and then written again, and so forth.

I have looked into one of the threads, and there is a rewriteDependencyReducedPomIfWeHaveReduction in ShadeMojo with a loopCounter with value 2735, that just keeps increasing. Presumably there is something like one dependency-reduced-pom.xml written per iteration.

From the source code it seems this can only happen if https://github.com/apache/maven-shade-plugin/blob/master/src/main/java/org/apache/maven/plugins/shade/mojo/ShadeMojo.java#L1172 is hit at least that number of times, meaning the ShadeMojo adds that many exclusions, which seems to correspond to hamcrest-core:

 <dependencies>
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
      <version>4.12</version>
     <scope>test</scope>
    <exclusions>
      <exclusion>
        <artifactId>hamcrest-core</artifactId>
        <groupId>org.hamcrest</groupId>
      </exclusion>
      <exclusion>
        <artifactId>hamcrest-core</artifactId>
        <groupId>org.hamcrest</groupId>
      </exclusion>
     ...

grep hamcrest-core dependency-reduced-pom.xml | wc -l
 2735

 

The same exclusion is added loopCounter times in updateExcludesInDeps.

There is no problem running mvn in single-thread mode.

---

Affects: 3.2.1

Attachments:

• mshade-329-thread-dump.txt (34.15 kB)

Issue Links:

• MSHADE-124 Need better plan for getting dependency-reduced-pom.xml out of basedir

• FLINK-14318 JDK11 build stalls during shading

1 votes, 5 watchers

[jira-importer]

Bruno Medeiros commented

There is no problem running mvn in single-thread mode. I can confirm this bug, same thing happening here if -T4C is added to the build.

I'm attaching a thread dump taken while maven build was stuck, we can seem some builder threads in RUNNABLE state, but writing to files like crazy.

As disabling parallel build seems a very harsh workaround, we are doing this here:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <configuration>
        <!-- disabling reduced pom generation due to https://issues.apache.org/jira/browse/MSHADE-329 -->
        <createDependencyReducedPom>false</createDependencyReducedPom>
    </configuration>
</plugin>

I even dare to suggest we disable reduced pom generation by default as most people using it are just generating a uber jar and don't need a pom for that (personal opinion).