DependencyCheck icon indicating copy to clipboard operation
DependencyCheck copied to clipboard

Published 20 hours ago • verified publisher icon jeremylong

Update Errors

Open Teicu opened this issue 1 year ago • 13 comments
trafficstars

Hello,

Any idea why I keep getting these errors? Thank you

[INFO] Checking for updates [INFO] NVD API has 242,601 records in this update [INFO] Downloaded 10,000/242,601 (4%) [INFO] Downloaded 20,000/242,601 (8%) [INFO] Downloaded 30,000/242,601 (12%) [INFO] Downloaded 40,000/242,601 (16%) [INFO] Downloaded 50,000/242,601 (21%) [INFO] Downloaded 60,000/242,601 (25%) [INFO] Downloaded 70,000/242,601 (29%) [INFO] Downloaded 80,000/242,601 (33%) [INFO] Downloaded 90,000/242,601 (37%) [INFO] Downloaded 100,000/242,601 (41%) [INFO] Downloaded 110,000/242,601 (45%) [INFO] Downloaded 120,000/242,601 (49%) [INFO] Downloaded 130,000/242,601 (54%) [INFO] Downloaded 140,000/242,601 (58%) [INFO] Downloaded 150,000/242,601 (62%) [INFO] Downloaded 160,000/242,601 (66%) [INFO] Downloaded 170,000/242,601 (70%) [INFO] Downloaded 180,000/242,601 (74%) [INFO] Downloaded 190,000/242,601 (78%) [INFO] Downloaded 200,000/242,601 (82%) [INFO] Downloaded 210,000/242,601 (87%) [INFO] Downloaded 220,000/242,601 (91%) [INFO] Downloaded 230,000/242,601 (95%) [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637) at org.owasp.dependencycheck.App.runScan(App.java:262) at org.owasp.dependencycheck.App.run(App.java:194) at org.owasp.dependencycheck.App.main(App.java:89) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349) ... 7 common frames omitted [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist

Teicu avatar Mar 25 '24 17:03 Teicu

[INFO] Checking for updates [INFO] NVD API has 242,601 records in this update [INFO] Downloaded 10,000/242,601 (4%) [INFO] Downloaded 20,000/242,601 (8%) [INFO] Downloaded 30,000/242,601 (12%) [INFO] Downloaded 40,000/242,601 (16%) [INFO] Downloaded 50,000/242,601 (21%) [INFO] Downloaded 60,000/242,601 (25%) [INFO] Downloaded 70,000/242,601 (29%) [INFO] Downloaded 80,000/242,601 (33%) [INFO] Downloaded 90,000/242,601 (37%) [INFO] Downloaded 100,000/242,601 (41%) [INFO] Downloaded 110,000/242,601 (45%) [INFO] Downloaded 120,000/242,601 (49%) [INFO] Downloaded 130,000/242,601 (54%) [INFO] Downloaded 140,000/242,601 (58%) [INFO] Downloaded 150,000/242,601 (62%) [INFO] Downloaded 160,000/242,601 (66%) [INFO] Downloaded 170,000/242,601 (70%) [INFO] Downloaded 180,000/242,601 (74%) [INFO] Downloaded 190,000/242,601 (78%) [INFO] Downloaded 200,000/242,601 (82%) [INFO] Downloaded 210,000/242,601 (87%) [INFO] Downloaded 220,000/242,601 (91%) [INFO] Downloaded 230,000/242,601 (95%) [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637) at org.owasp.dependencycheck.App.runScan(App.java:262) at org.owasp.dependencycheck.App.run(App.java:194) at org.owasp.dependencycheck.App.main(App.java:89) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349) ... 7 common frames omitted [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist

Teicu avatar Mar 25 '24 18:03 Teicu

Hey ! I am getting the same error

[INFO] Checking for updates [INFO] NVD API has 242.601 records in this update [INFO] Downloaded 10.000/242.601 (4%) [INFO] Downloaded 20.000/242.601 (8%) [INFO] Downloaded 30.000/242.601 (12%) [INFO] Downloaded 40.000/242.601 (16%) [INFO] Downloaded 50.000/242.601 (21%) [INFO] Downloaded 60.000/242.601 (25%) [INFO] Downloaded 70.000/242.601 (29%) [INFO] Downloaded 80.000/242.601 (33%) [INFO] Downloaded 90.000/242.601 (37%) [INFO] Downloaded 100.000/242.601 (41%) [INFO] Downloaded 110.000/242.601 (45%) [INFO] Downloaded 120.000/242.601 (49%) [INFO] Downloaded 130.000/242.601 (54%) [INFO] Downloaded 140.000/242.601 (58%) [INFO] Downloaded 150.000/242.601 (62%) [INFO] Downloaded 160.000/242.601 (66%) [INFO] Downloaded 170.000/242.601 (70%) [INFO] Downloaded 180.000/242.601 (74%) [INFO] Downloaded 190.000/242.601 (78%) [INFO] Downloaded 200.000/242.601 (82%) [INFO] Downloaded 210.000/242.601 (87%) [INFO] Downloaded 220.000/242.601 (91%) [INFO] Downloaded 230.000/242.601 (95%) [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637) at org.owasp.dependencycheck.App.runScan(App.java:262) at org.owasp.dependencycheck.App.run(App.java:194) at org.owasp.dependencycheck.App.main(App.java:89) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349) ... 7 common frames omitted [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist

Any idea?

crizulm avatar Mar 25 '24 18:03 crizulm

This is happening practically daily and apparently no solution has been made available yet.

nicollasteixeira avatar Mar 25 '24 20:03 nicollasteixeira

this looks similar to #6547 ?

defaultbranch avatar Mar 25 '24 21:03 defaultbranch

i keep getting this isuue logs on console [WARN] NVD API request failures are occurring; retrying request for the 10 time

shekharclover avatar Mar 26 '24 06:03 shekharclover

Apparently the NVD API - which is not controlled by this project is having issues. Not much I can do.

jeremylong avatar Mar 26 '24 10:03 jeremylong

Apparently the NVD API - which is not controlled by this project is having issues. Not much I can do.

Hey Jeremy, appreciate the heads up. By the way, is there any alternative method for updating? I'm keen on using this tool, I've been struggling for the past two days trying to scan a local file.

Cheers

Teicu avatar Mar 26 '24 10:03 Teicu

  • Keep a copy of the DB after you create it (some actually just rebuild every scan which is horrifying): https://jeremylong.github.io/DependencyCheck/data/cacheh2.html
  • Create and use a mirror of the NVD: https://jeremylong.github.io/DependencyCheck/data/mirrornvd.html
    • there is a mirror https://dependency-check.github.io/DependencyCheck_Builder/

jeremylong avatar Mar 26 '24 11:03 jeremylong

The strange thing is that if I do it inside a docker (linux) it breaks, but if I run in my MacOS machine, it works.

[INFO] Download Started for NVD CVE - Modified [INFO] Download Complete for NVD CVE - Modified (7130 ms) [INFO] Processing Started for NVD CVE - Modified [INFO] Processing Complete for NVD CVE - Modified (3873 ms) [INFO] Begin database maintenance [INFO] Updated the CPE ecosystem on 133015 NVD records [INFO] End database maintenance (8264 ms) [INFO] Skipping RetireJS update since last update was within 24 hours. [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [INFO] Begin database defrag [INFO] End database defrag (3642 ms) [INFO] Check for updates complete (32053 ms) [INFO]

tarrinho avatar Mar 26 '24 11:03 tarrinho

The strange thing is that if I do it inside a docker (linux) it breaks, but if I run in my MacOS machine, it works.

[INFO] Download Started for NVD CVE - Modified [INFO] Download Complete for NVD CVE - Modified (7130 ms) [INFO] Processing Started for NVD CVE - Modified [INFO] Processing Complete for NVD CVE - Modified (3873 ms) [INFO] Begin database maintenance [INFO] Updated the CPE ecosystem on 133015 NVD records [INFO] End database maintenance (8264 ms) [INFO] Skipping RetireJS update since last update was within 24 hours. [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [INFO] Begin database defrag [INFO] End database defrag (3642 ms) [INFO] Check for updates complete (32053 ms) [INFO]

that's pretty weird, I have this problem with my Mac

Teicu avatar Mar 26 '24 11:03 Teicu

@jeremylong Thanks for letting us know. I saw your comment about caching nvd data but how do I point out to dependency check that it should use these local vulnerabilities? Is it possible to download the vulnerabilities and whenever a scan is done it is based on this local database?

nicollasteixeira avatar Mar 26 '24 13:03 nicollasteixeira

Investigating this further, it appears to relate to this issue

I am having the same error where the client attempts to fetch the same index 5 times then gives up. If I use curl to download the index that it is having trouble with, then manually parse the json with objectMapper.readValue(json, CveApiJson20::class.java), I get the exception: Unrecognized field "cvssMetricV40" (class io.github.jeremylong.openvulnerability.client.nvd.Metrics), not marked as ignorable (3 known properties: "cvssMetricV30", "cvssMetricV31", "cvssMetricV2"]) The JSON I downloaded from NVD does indeed contain a cvssMetricV40 property.

It appears that any JSON parsing error in the client results in just retrying the download again until it gives up after 5 tries.

The issue linked has a PR to ignore any unknown properties in the JSON which would likely resolve this issue too.

vptp avatar Mar 26 '24 22:03 vptp

I just merged https://github.com/jeremylong/DependencyCheck/pull/6554 - so if people are having an issue due to the cvssMetricsV40 - that will be fixed with the next release.

jeremylong avatar Mar 31 '24 11:03 jeremylong