Could not perform Node Audit analysis

[sranney]

My team has been working over the last couple of days to understand why this error is being thrown in our Jenkins build pipeline:

[2021-10-08T18:09:32.862Z] [DependencyCheck] [INFO] Checking for updates
[2021-10-08T18:09:32.869Z] [DependencyCheck] [INFO] Skipping NVD check since last check was within 4 hours.
[2021-10-08T18:09:32.958Z] [DependencyCheck] [INFO] Check for updates complete (96 ms)
[2021-10-08T18:09:33.112Z] [DependencyCheck] [INFO] 
[2021-10-08T18:09:33.112Z] [DependencyCheck] 
[2021-10-08T18:09:33.112Z] [DependencyCheck] Dependency-Check is an open source tool performing a best effort analysis of 3rd party dependencies; false positives and false negatives may exist in the analysis performed by the tool. Use of the tool and the reporting provided constitutes acceptance for use in an AS IS condition, and there are NO warranties, implied or otherwise, with regard to the analysis or its use. Any use of the tool and the reporting provided is at the user’s risk. In no event shall the copyright holder or OWASP be held liable for any damages whatsoever arising out of or in connection with the use of this tool, the analysis performed, or the resulting report.
[2021-10-08T18:09:33.112Z] [DependencyCheck] 
[2021-10-08T18:09:33.112Z] [DependencyCheck] 
[2021-10-08T18:09:33.112Z] [DependencyCheck]    About ODC: https://jeremylong.github.io/DependencyCheck/general/internals.html
[2021-10-08T18:09:33.112Z] [DependencyCheck]    False Positives: https://jeremylong.github.io/DependencyCheck/general/suppression.html
[2021-10-08T18:09:33.112Z] [DependencyCheck] 
[2021-10-08T18:09:33.112Z] [DependencyCheck] 💖 Sponsor: https://github.com/sponsors/jeremylong
[2021-10-08T18:09:33.112Z] [DependencyCheck] 
[2021-10-08T18:09:33.112Z] [DependencyCheck] 
[2021-10-08T18:09:33.112Z] [DependencyCheck] [INFO] Analysis Started
[2021-10-08T18:09:33.117Z] [DependencyCheck] [INFO] Finished Archive Analyzer (0 seconds)
[2021-10-08T18:09:33.125Z] [DependencyCheck] [INFO] Finished File Name Analyzer (0 seconds)
[2021-10-08T18:09:33.127Z] [DependencyCheck] [INFO] Finished Dependency Merging Analyzer (0 seconds)
[2021-10-08T18:09:33.131Z] [DependencyCheck] [INFO] Finished Version Filter Analyzer (0 seconds)
[2021-10-08T18:09:33.239Z] [DependencyCheck] [INFO] Finished Hint Analyzer (0 seconds)
[2021-10-08T18:09:33.245Z] [DependencyCheck] [INFO] Finished False Positive Analyzer (0 seconds)
[2021-10-08T18:09:33.248Z] [DependencyCheck] [INFO] Finished NVD CVE Analyzer (0 seconds)
[2021-10-08T18:09:33.250Z] [DependencyCheck] [WARN] The Node Package Analyzer has been disabled; the resulting report will only contain the known vulnerable dependency - not a bill of materials for the node project.
[2021-10-08T18:09:33.687Z] [DependencyCheck] [WARN] dependency skipped: package.json contain an alias for vue-loader-v16 => [email protected] npm audit doesn't support aliases
[2021-10-08T18:09:34.336Z] [DependencyCheck] [ERROR] NodeAuditAnalyzer failed on /var/lib/jenkins/workspace/NL_ui-components_PR-280/package-lock.json
[2021-10-08T18:09:34.336Z] [DependencyCheck] [WARN] An error occurred while analyzing '/var/lib/jenkins/workspace/NL_ui-components_PR-280/package-lock.json' (Node Audit Analyzer).
[2021-10-08T18:09:34.336Z] [DependencyCheck] [INFO] Finished Node Audit Analyzer (1 seconds)
[2021-10-08T18:09:34.337Z] [DependencyCheck] [INFO] Finished Sonatype OSS Index Analyzer (0 seconds)
[2021-10-08T18:09:34.465Z] [DependencyCheck] [INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
[2021-10-08T18:09:34.465Z] [DependencyCheck] [INFO] Finished Dependency Bundling Analyzer (0 seconds)
[2021-10-08T18:09:34.477Z] [DependencyCheck] [INFO] Analysis Complete (1 seconds)
[2021-10-08T18:09:34.581Z] [DependencyCheck] [INFO] Writing report to: /var/lib/jenkins/workspace/NL_ui-components_PR-280/analysis/dependency-check-report.xml
[2021-10-08T18:09:34.719Z] [DependencyCheck] [INFO] Writing report to: /var/lib/jenkins/workspace/NL_ui-components_PR-280/analysis/dependency-check-report.html
[2021-10-08T18:09:35.107Z] [DependencyCheck] [ERROR] Could not perform Node Audit analysis. Invalid payload submitted to Node Audit API.

I have attached the package.json file and the package-lock.json file for this project in the issue. Can you give me some guidance on what could be causing the above issue, and how to fix it, given this information? dependency check attachments - package and package-lock.zip

[sranney]

wanted to bump this, in hopes we can get some help. Thanks!

[rlucha]

We are observing the same failure, and so far it looks flaky. We cannot see anything in the affected package.json or package-lock.json that could trigger this error.

[Jakeb98]

Also getting lots of these errors when trying to implement Dependency-Check into our project. We have a fairly large project with many sub-projects that each have their own package-lock.json, and it works fine for many and gives the above error for others. We haven't been able to find anything in the failing ones that differentiates them from the passing ones to cause the error.

[jeremylong]

Pretty sure this will be fixed in 6.5.0 which was slightly delayed but should be released this weekend.

[jonnybbb]

@jeremylong it doesn't seem to be fixed for us. Can you point me to the code change that supposed to fix the problem. What information do you need to get the insides you need to better understand the issue we face?

[racedale]

6.5.0 did not fix this for us. Seems to be due to package-locks generated with newer versions of npm (7, 8)

[StudentBlake]

Can confirm, it’s totally broken on package-lock.json files generated with latest stable version of npm (8). For us, this is on multiple projects, not sure if it’s every case. This has been an issue for a while, hoping it can get a bump in priority.

[Samuel-Schober-USU]

We have the same issue with package-lock.json files generated by npm 8, so many of our CI/CD pipelines are breaking. The current workaround to allow the dependency scan job to fail is not really a good approach, so we would also love if the priority of this issue could be bumped.

Just an idea, but maybe the change from lockfileVersion 1 to lockfileVersion 2 causes some troubles? https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json#lockfileversion

[oliverlockwood]

Interestingly, I'm finding that:

• if I force usage of Node 14 (and hence NPM version 6.4.15, at the time of writing) this issue does not occur.
• if I force usage of Node 12 (and hence NPM version 6.4.15, at the time of writing) this issue does not occur.
• if I force usage of Node 10 (and hence NPM version 6.14.12, at the time of writing) this issue does not occur.

Whereas:

• if I force usage of Node 16 (and hence NPM version 8.1.2, at the time of writing) this issue occurs consistently.

So this seems to confirm @racedale, @StudentBlake and @Samuel-Schober-USU's suspicions; and makes it clear that there is a sort-of-workaround for now (i.e. force Node 14 and regenerate package-lock.json), though it is clearly a bit ugly for any projects that are using NodeJS >14.

Hopefully @jeremylong this is helpful to you in narrowing down the issue. Perhaps you can experiment locally with NodeJS 16 and thereby identify the root of the problem.

Oh, and I upgraded to the very latest Dependency-Check 6.5.3 - same behaviour.

[diogofscmariano]

Same thing happening here. CI/CD are failing with this error and the move was to live with the existing findings. Project relying on Node 14, and NodeJS Audit checks are failing. Any update?

[kascaks]

I had the same problem.

In my case the situation was that on dev machine node14/npm6 was installed which generated package-lock.json with lockfileVersion 1 that was commited to git (node_modules are not commited) from which it was picked up by CI with node16/npm8. DependencyCheck was running fine on dev, but failing on CI.

After some try-and-error I solved it by removing node14/npm6, deleting node_modules and package-lock.json then installing node16/npm8 and recreating node_modules and package-lock.json (by npm install) on dev and commited new package-lock.json which now had lockfileVersion 2. DependencyCheck now runs fine on both dev and CI.

It's also probably worth mentioning that I'm running DependencyCheck as maven plugin 7.0.0

[vidal7]

So, I understand that we cannot use DependencyCheck anymore with Node 16+ or npm 7/8 and a package-lock.json with lockFileVersion 2 ?

[ryandutton]

That's correct, the payload which is sent to the NPM API is invalid resulting in an error being returned by the API. The npm audit cli command against the same project with the same package-lock.json file will succeed.

[Janpopan]

There is an other ticket around this checker (and the problem ? ) https://github.com/jeremylong/DependencyCheck/issues/4536

[mum-viadee]

I made an interesting observation. I took the submitted payload from the logfile and posted it using curl in a shell. Surprisingly it worked and I got an HTTP status 200 and a valid response. Then I tried it with a small Java program that uses code snippets from the owasp dependency check. That didn't work. The difference I noticed, after looking at the curl request in detail was the content-type header. Curl, per default, posts files as "application/x-www-form-urlendoded". After changing the code to set the same header as curl does, it worked. Changing the header in curl got me a 400 Bad request likewise.

It looks like a strange behaviour to me, does anyone have any documentation for die npm audit api?

[ryandutton]

I've been doing a bit of digging and one of the things I found was that the content-type needs to be set as application/json if running using curl, as suggested by @mum-viadee. The second thing I found was that since npm v7 the npm/cli project uses Arborist to generate the dependency tree. It looks like it then submits the tree to the /-/npm/v1/security/advisories/bulk endpoint, exposing the report inside the npm/cli codebase. As Dependency Check is Java it makes this much more complicated to fix as it will have to essentially mirror the structuring that arborist does.

[itsecforu]

the same isuue with:

Dependency-Check Core version 7.1.1

if i grep "error" depcheck.log:

2022-06-28 16:08:29,482 org.owasp.dependencycheck.App:211
DEBUG - unexpected [32;102;21Merror
org.owasp.dependencycheck.analyzer.exception.SearchException: Could not perform Node Audit analysis. Invalid payload submitted to Node Audit API.
<------>at org.owasp.dependencycheck.data.nodeaudit.NodeAuditSearch.submitPackage(NodeAuditSearch.java:209)
<------>at org.owasp.dependencycheck.data.nodeaudit.NodeAuditSearch.submitPackage(NodeAuditSearch.java:133)
<------>at org.owasp.dependencycheck.analyzer.NodeAuditAnalyzer.analyzePackage(NodeAuditAnalyzer.java:189)
<------>at org.owasp.dependencycheck.analyzer.NodeAuditAnalyzer.analyzeDependency(NodeAuditAnalyzer.java:146)
<------>at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
<------>at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
<------>at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
<------>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
<------>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
<------>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
<------>at java.lang.Thread.run(Thread.java:748)
2022-06-28 16:08:29,482 org.owasp.dependencycheck.utils.Settings:821
DEBUG - Deleting ALL temporary files from `/tmp/dctempf41abb9a-4037-44d1-bb85-775cf18c273c`
2022-06-28 16:08:29,520 org.owasp.dependencycheck.App:84
DEBUG - Exit code: -14

could the fact that the package-lock.json file is 2.5 megabytes be a problem?

[Benoit-Vasseur]

Hello,

Is there a fix planned and or identified :) ? We've just upgraded our npm version on CI (v6 -> v8) and it now fail on some projects. (Node16 is the new LTS)

[v1shva]

I had the same problem.

In my case the situation was that on dev machine node14/npm6 was installed which generated package-lock.json with lockfileVersion 1 that was commited to git (node_modules are not commited) from which it was picked up by CI with node16/npm8. DependencyCheck was running fine on dev, but failing on CI.

After some try-and-error I solved it by removing node14/npm6, deleting node_modules and package-lock.json then installing node16/npm8 and recreating node_modules and package-lock.json (by npm install) on dev and commited new package-lock.json which now had lockfileVersion 2. DependencyCheck now runs fine on both dev and CI.

It's also probably worth mentioning that I'm running DependencyCheck as maven plugin 7.0.0

We already had our package-lock.json upgraded to "lockfileVersion": 2 still it was failing. I removed the package-lock.json and node_modules folder and did a npm install (in node 16.14.0) and it did fix the issue.

[alex-arzaghi]

Pretty sure this will be fixed in 6.5.0 which was slightly delayed but should be released this weekend.

@jeremylong Has this version/fix been releases?

[oliverlockwood]

@alex-arzaghi FYI in our pipelines, we're still using Node 14 specifically for Dependency-Check, due to this issue 😞

[delenikov]

Check this https://github.com/jeremylong/DependencyCheck/issues/3716#issuecomment-1550860049

[StudentBlake]

Check this #3716 (comment)

Is there a way to specify the endpoint for the cli? I don't see that param in the documentation.

[itineric]

The option does not exist on command line but can be provided using the properties file.

Add the following to your existing properties file analyzer.node.audit.url=/-/npm/v1/security/advisories/bulk

And if not already using a properties file, create one and then add -P my-config.properties to the command line

[paul-dyson]

What seemed to work for me was to reinstate the resolved and integrity values in package-lock.json that npm v8 had lost by running

npx [email protected] install && npm install

After which the previously failing Node Audit analysis completed without error.

[adcorduneanu]

This still happens in 8.2.1, with node 18.x.

[fascynacja]

Did anyone found a workaround for this issue? I am facing same problem with command:

mvn org.owasp:dependency-check-maven:7.3.2:aggregate and node version: 14.x.x