Version 8.0.0. KnownExploitedDataSource proxy HTTP 403

[marwin1991]

KnownExploitedDataSource Line: 78

https://github.com/jeremylong/DependencyCheck/blob/f3d806de22002d932fe12a706f4b9fa91ed3322d/core/src/main/java/org/owasp/dependencycheck/data/update/KnownExploitedDataSource.java

                //TODO - add all the proxy config, likely use the same as configured for NVD
                final HttpResourceConnection conn = new HttpResourceConnection(settings);

In class HostedSuppressionsDataSource i do not see such a commonet:

https://github.com/jeremylong/DependencyCheck/blob/f3d806de22002d932fe12a706f4b9fa91ed3322d/core/src/main/java/org/owasp/dependencycheck/data/update/HostedSuppressionsDataSource.java

Logs:

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:8.0.0:check (default-cli) on project iaml: Fatal exception(s) analyzing aaaaaaaaaa: One or more exceptions occurred during analysis:
[ERROR] 	UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR] 		caused by DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR] 		caused by IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"
[ERROR] 	NoDataException: No documents exist
[ERROR] -> [Help 1]
[ERROR] 

[lukasT]

I got a similar problem (getting 403 when trying to download the external file) which might be related, if the plugin was executed within a gitlab-ci pipeline job:

[ERROR] 	UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR] 		caused by DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR] 		caused by DownloadFailedException: Error retrieving https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; received response code 403; Forbidden
[ERROR] 	NoDataException: No documents exist

so maybe our problem is the same that somehow the https://www.cisa.gov/ page does not allow your proxy and my direct download...not sure, why though

[jeremylong]

As @aikebah noted in PR #5330- the proxy configuration is already being used if it is configured.

Are there rules setup to allow the NVD that would need to be replicated in your proxy for cisa.gov?

[marwin1991]

As @aikebah noted in PR #5330- the proxy configuration is already being used if it is configured.

Are there rules setup to allow the NVD that would need to be replicated in your proxy for cisa.gov?

DevOps team has changed something in the configuration to fix this, but error as mentioned before occured while executing command:

      mvn $COMPILE \
      --batch-mode \
      -Dhttp.proxyHost=X.X.X.X \
      -Dhttps.proxyHost=X.X.X.X \
      -Dhttp.proxyPort=8080 \
      -Dhttps.proxyPort=8080 \
      -Dhttp.nonProxyHosts="a.b.c" \
      -Dhttps.nonProxyHosts="*.a.b.c" \
      $SECURITY_DEPENDENCY_SCANNING_MAVEN_EXTRA_ARGS \
      org.owasp:dependency-check-maven:check -Dformat=JUNIT

[aikebah]

DevOps team has changed something in the configuration to fix this

My gut feel: they whitelisted the URL or the entire CISA website on your proxy

[marwin1991]

Hello, what about version 8.0.1 ? It's not using proxy to get publishedSuppressions.xml

Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection (HttpResourceConnection.java:267)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch (HttpResourceConnection.java:163)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:130)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:74)
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions (HostedSuppressionsDataSource.java:133)
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.update (HostedSuppressionsDataSource.java:77)
    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:900)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:705)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:631)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1914)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1096)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke (Method.java:578)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"
    at sun.net.www.protocol.http.HttpURLConnection.doTunneling0 (HttpURLConnection.java:2262)
    ```

[aikebah]

Well... still it IS using your proxy, but your proxy is returning

Proxy returns "HTTP/1.1 403 Forbidden

so you need to find out why your proxy is denying you access to the CISA known exploited vulnerabilities

[aikebah]

most likely you run it in a network-access restricted environment with a proxy that has whitelisted "allowed locations" and the CISA website is not among them, so contact your network team

[marwin1991]

Tank you so much @aikebah

[sanddorn]

Is there a known bann-list on cisa.gov for the hetzner network? I get a 403 from cisa.gov on all of my hetzner servers (cloud and root-server), but not on digitalocean or on the telekom.de network via DSL.

[lukasT]

@sanddorn that would explain my problem too as we have gitlab runners on hetzner.

[sanddorn]

@lukasT You can simply check by a curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json That runs on most systems, but on Hetzner. That's part of my question. It seems, there es a ban on the hetzner networks.

On Hetzer, I get an 403, 200 and lots of data anywhere else.

[sanddorn]

@lukasT Changing Runner to Hillsboro (US) solves the problem.

[thomseno]

@lukasT You can simply check by a curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json That runs on most systems, but on Hetzner. That's part of my question. It seems, there es a ban on the hetzner networks.

On Hetzer, I get an 403, 200 and lots of data anywhere else.

Experiencing the same problem... gitlab-runner on Hetzner, scan is failing because not able to download from cisa.gov. Downgraded to maven-plugin version 7.4.4.

[sblack4]

I'm also getting a 403 from a runner on the Hetzner network. I can't find anything about a ban list, though.

[jeremylong]

The options would be to mirror the catalog or disable the analyzer.

[thomseno]

The options would be to mirror the catalog or disable the analyzer.

How to disable the Known Exploited Vulnerability analyzer from Maven plugin?

[thomseno]

The options would be to mirror the catalog or disable the analyzer.

How to disable the Known Exploited Vulnerability analyzer from Maven plugin?

Forget it... found it: -DknownExploitedEnabled=false

[aikebah]

Forget it... found it: -DknownExploitedEnabled=false

Either like that, or by specifying <knownExploitedEnabled>false</knownExploitedEnabled> in the plugin configuration section in your pom-file

[alan-czajkowski]

you can use a mirror or proxy via the plugin config like so:

<knownExploitedUrl>https://your-mirror-or-proxy/repository/cisa-files/sites/default/files/feeds/known_exploited_vulnerabilities.json</knownExploitedUrl>

[alan-czajkowski]

@jeremylong you should probably update the docs to make it more obvious on how to configure this stuff

[UbiNickUfer]

The site is blocked in GCP europe-west too. Can't tell if it blocked the whole GCP network but this is already pretty bad on its own. Why are they doing this? Do they want that nobody is using this?

[msaubier]

My team has been running into this issue intermittently this week...sometimes it works, sometimes it doesn't when attempting to download the CISA feed in our AWS CodePipeline. Is there a drawback to disabling the CISA Known Exploited Vulnerability analyzer? Would we potentially miss scanning for any CVEs?

[jeremylong]

The known exploited vulnerability catalog does not add any new vulnerabilities... but any vulnerabilities in the catalog likely need to be patched ASAP as there are known attacks happening using the CVE.

On Thu, Jun 29, 2023, 2:52 PM msaubier @.***> wrote:

My team has been running into this issue intermittently this week...sometimes it works, sometimes it doesn't. Is there a drawback to disabling the CISA Known Exploited Vulnerability analyzer? Would we potentially miss scanning for any CVEs?

— Reply to this email directly, view it on GitHub https://github.com/jeremylong/DependencyCheck/issues/5313#issuecomment-1613635432, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGSVQQYBZFMDYPOZOMGEA3XNXFGLANCNFSM6AAAAAAT4SQELM . You are receiving this because you were mentioned.Message ID: @.***>

[lgolubenkobit]

This week we are getting this error

[INFO] --- dependency-check:9.1.0:check (default) @ canvas-root ---
[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 480 minutes
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[ERROR] Error retrieving https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; received response code 403; Forbidden
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
org.owasp.dependencycheck.data.update.exception.UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
    at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update (KnownExploitedDataSource.java:93)

and after that we cannot analyze any project by some time.

We are using apiKey and this error is only reported with known_exploited_vulnerabilities.json, previous update is done ok.

If i try to do curl it works ok. curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

We had the same problem with version 7. About 1 month ago we updated dependency-check from 7 to 9 and works ok without problems on more than 30 pipelines that sometimes can run at same time, until this week that on some moments all pipelines running fail.

[lgolubenkobit]

This week we are getting this error

[INFO] --- dependency-check:9.1.0:check (default) @ canvas-root ---
[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 480 minutes
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[ERROR] Error retrieving https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; received response code 403; Forbidden
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
org.owasp.dependencycheck.data.update.exception.UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
    at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update (KnownExploitedDataSource.java:93)

and after that we cannot analyze any project by some time.

We are using apiKey and this error is only reported with known_exploited_vulnerabilities.json, previous update is done ok.

If i try to do curl it works ok. curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

We had the same problem with version 7. About 1 month ago we updated dependency-check from 7 to 9 and works ok without problems on more than 30 pipelines that sometimes can run at same time, until this week that on some moments all pipelines running fail.

Our pipelines run on AWS.

2024-05-13T16:15:51 First pipeline running on the day, but according last pipeline runned 4 days ago, then update and works ok.

2024-05-13T17:27:51, about one hour later, in the same repository (using cache for .m2 folder, then nothing should be updated), but dependency list is updated and then get error

2024-05-13T17:42:04, 10 minutes later, after pipeline retry, works ok

There aren't other pipelines running at same time (using the same key).

[aikebah]

2024-05-13T17:27:51, about one hour later, in the same repository (using cache for .m2 folder, then nothing should be updated), but dependency list is updated and then get error

Default validForHours of NVD API data is 4hrs, but updates were requested, so it looks like you were using a different .m2 cache in that run. KEV is using a validity period of 24hrs by default, so again indicating that you likely got a different .m2 cache for that run.

[lgolubenkobit]

2024-05-13T17:27:51, about one hour later, in the same repository (using cache for .m2 folder, then nothing should be updated), but dependency list is updated and then get error

Default validForHours of NVD API data is 4hrs, but updates were requested, so it looks like you were using a different .m2 cache in that run. KEV is using a validity period of 24hrs by default, so again indicating that you likely got a different .m2 cache for that run.

Assuming it could be an error that the same .m2 directory is not being used in the same repo, why do we get the http 403 forbidden error without having exceeded the usage rate when using apiKey?

What causes a http 403 updating CISA Known Exploited Vulnerability list?

[aikebah]

What causes it you should ask the CISA.

In any case the NVD apiKey is for access to NIST NVD CVE-API and has no effect or usage with the CISA KEV json downloads (they are different resources operated by different organisations)

Regarding the ways in which you could have multiple .m2 caches: that would fully depend on how your AWS cloud build pipelines environment is operated.

E.g. I can imagine setups with caches to be datacenter-, serverrack- or even buildrunner-local rather than a global share that is replicated introducing synchronization latencies or a globally shared single share introducing continuous I/O latencies for any non-local build-runner host (and unavailability in case of single system outage).

[lgolubenkobit]

What causes it you should ask the CISA.

In any case the NVD apiKey is for access to NIST NVD CVE-API and has no effect or usage with the CISA KEV json downloads (they are different resources operated by different organisations)

Regarding the ways in which you could have multiple .m2 caches: that would fully depend on how your AWS cloud build pipelines environment is operated.

E.g. I can imagine setups with caches to be datacenter-, serverrack- or even buildrunner-local rather than a global share that is replicated introducing synchronization latencies or a globally shared single share introducing continuous I/O latencies for any non-local build-runner host (and unavailability in case of single system outage).

Thanks for the clarification. I thought the same apiKey was used for both downloads. At cache level we didn't found any error.

Now we are disabling cisa kve download by pom propertie and later we are going to mirror it locally.

[gpt8763]

use this method can solve problem vi dependency-check.sh exec "$JAVACMD" $JAVA_OPTS $DEBUG
-classpath "$CLASSPATH"
-Dapp.name="dependency-check"
-Dapp.pid="$$"
-Dapp.repo="$REPO"
-Dapp.home="$BASEDIR"
-Dbasedir="$BASEDIR"
-Dkev.url=file:///known_exploited_vulnerabilities.json
org.owasp.dependencycheck.App
"$@"

download a file known_exploited_vulnerabilities.json on chrome https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json