`ConcurrentModificationException` in `NodePackageAnalyzer.processDependencies`

[pshevche]

trafficstars

Describe the bug ConcurrentModificationException in NodePackageAnalyzer.processDependencies. Seems like a regression of https://github.com/jeremylong/DependencyCheck/issues/4089

Version of dependency-check used 8.1.0 version of the Gradle plugin, but I see that it was also happening as early as 7.0.1

Log file

An unexpected error occurred during analysis of '.../package-lock.json' (Node.js Package Analyzer): null

java.util.ConcurrentModificationException
	at java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1208)
	at java.base/java.util.TreeMap$KeyIterator.next(TreeMap.java:1262)
	at java.base/java.lang.Iterable.forEach(Iterable.java:74)
	at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
	at org.owasp.dependencycheck.analyzer.DependencyMergingAnalyzer.mergeDependencies(DependencyMergingAnalyzer.java:160)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:465)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:410)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.analyzeDependency(NodePackageAnalyzer.java:270)
	at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
	at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
	at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

To Reproduce Can't provide clear reproduction steps, the number of exceptions and in which projects they occur is pretty random.

Expected behavior No ConcurrentModificationException and analysis is complete

Additional context

[jeremylong]

@aikebah do you see why this is happening?

[aikebah]

@jeremylong at first glance I could only logically imagine it happening if dependency == relatedDependency when the ConcurrentModificationException happens. I would not expect this to ever hold true for mergeDependencies, but I'll dive a bit more in the codebase to see if I can dig up some logic error that might trigger just that.

[kevintap]

We seem to be able to recreate this when analyzing a mono repo nodejs project. It appears that a single sub-project with a node_modules directory works ok, but if you npm install additional sub-projects, we get the exception.

[ERROR] java.util.ConcurrentModificationException: null at java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1486) at java.base/java.util.TreeMap$KeyIterator.next(TreeMap.java:1540) at java.base/java.lang.Iterable.forEach(Iterable.java:74) at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1092) at org.owasp.dependencycheck.analyzer.DependencyMergingAnalyzer.mergeDependencies(DependencyMergingAnalyzer.java:160) at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:465) at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:410) at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.analyzeDependency(NodePackageAnalyzer.java:270) at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131) at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88) at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833)

[kevintap]

@aikebah @jeremylong Here's some additional detail from a debug log I collected on this issue.

DEBUG - Merging '/builds/my-components-1/package-lock.json?my-components-1:0.0.42/@babel/plugin-syntax-top-level-await:7.14.5' into '/builds/my-components-2/package-lock.json?my-components-2:0.0.101/@babel/plugin-syntax-top-level-await:7.14.5'
2023-04-06 14:43:15,711 org.owasp.dependencycheck.AnalysisTask:96
ERROR - 
java.util.ConcurrentModificationException: null
	at java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1486)
	at java.base/java.util.TreeMap$KeyIterator.next(TreeMap.java:1540)
	at java.base/java.lang.Iterable.forEach(Iterable.java:74)
	at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1092)
	at org.owasp.dependencycheck.analyzer.DependencyMergingAnalyzer.mergeDependencies(DependencyMergingAnalyzer.java:160)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:465)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:410)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.analyzeDependency(NodePackageAnalyzer.java:270)
	at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
	at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
	at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)




DEBUG - Merging '/builds/my-components-3/package-lock.json?my-components-3:0.0.4/@jest/schemas:29.4.0/@sinclair/typebox:^0.25.16' into '/builds/my-components-3/package-lock.json?my-components-3:0.0.4/@sinclair/typebox:0.25.21'
2023-04-06 14:43:15,657 org.owasp.dependencycheck.AnalysisTask:96
ERROR - 
java.util.ConcurrentModificationException: null
	at java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1486)
	at java.base/java.util.TreeMap$KeyIterator.next(TreeMap.java:1540)
	at java.base/java.lang.Iterable.forEach(Iterable.java:74)
	at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1092)
	at org.owasp.dependencycheck.analyzer.DependencyMergingAnalyzer.mergeDependencies(DependencyMergingAnalyzer.java:160)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:465)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.processDependencies(NodePackageAnalyzer.java:410)
	at org.owasp.dependencycheck.analyzer.NodePackageAnalyzer.analyzeDependency(NodePackageAnalyzer.java:270)
	at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
	at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
	at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

As mentioned in my previous post, we're running dependency-check on a mono repo which contains multiple projects (e.g. my-components-1, my-components-2, my-components-3).

In the logs above, it looks like one instance of the exception occurs because of a common dependency between 2 components. The second instance of the exception is occurring within a single component project in the repo.

[m1cm1c]

this happens on every other run when i specify --scan twice. i specify it to --scan '/absolute/path/**/package.json' --scan '/absolute/path/**/package-lock.json'

[dabal]

We were facing this issue as well. For us the solution provided in the pr solved the issue. https://github.com/jeremylong/DependencyCheck/pull/6501