Error CERTIFICATE_VERIFY_FAILED

[dersteppenwolf]

The following error appears for some of our services:

2019-09-20 02:10:40,371 - GeoHealthCheck.views - WARNING - Cannot expand plugin vars for GeoHealthCheck.plugins.probe.wms.WmsGetMapV1All err=SSLError(MaxRetryError("HTTPSConnectionPool(host='mydomain.com', port=443): Max retries exceeded with url: /geoserver/ows?service=WMS&request=GetCapabilities&version=1.1.1 (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)'),))",),)

Looks like the certificate is provided by an intermediary entity (RapidSSL ) and it isn't recognized by the http middleware. (A similar error appears in Qgis)

Root CA: CN = DigiCert Global Root CA OU = www.digicert.com O = DigiCert Inc C = US

Intermediate: CN = RapidSSL RSA CA 2018 OU = www.digicert.com O = DigiCert Inc C = US

[justb4]

Can't reproduce without a URL/domain name (guess it is not mydomain.com?). I advise to analyse the SSL certificate. I always use https://www.ssllabs.com/ssltest, e.g. https://www.ssllabs.com/ssltest/analyze.html?d=geohealthcheck.org&hideResults=on&latest . It takes some time, but provides an extensive report.

It could also be due to your Python version and underlying SSL library versions. We need more context for that.

[dersteppenwolf]

Can't reproduce without a URL/domain name

You can reproduce the issue with the following FeatureServer: https://gis3.igac.gov.co/arcgis/rest/services/Catastro/Catastro_service_2019_05/FeatureServer

For other applications using python we used this to avoid that problem: https://stackoverflow.com/questions/15445981/how-do-i-disable-the-security-certificate-check-in-python-requests

[justb4]

Almost sure this issue: https://medium.com/@moreless/how-to-fix-python-ssl-certificate-verify-failed-97772d9dd14c

Could reproduce with https://gis3.igac.gov.co/arcgis/rest/services/Catastro/Catastro_service_2019_05/FeatureServer and fix with setting export PYTHONHTTPSVERIFY=0 before running GHC. Working on Py2-only clean code-based solution.

[dersteppenwolf]

Hi, any ideas about how to implement that change for the app deployed as a docker container?

looks like adding ENV PYTHONHTTPSVERIFY=0 to the docker file isn't enough.

[jpulles]

Hi, we had a similar error message, but the problem was that we had not installed the certificate files correctly. There was no problem with GeoHealthCheck. After fixing the certificate files, it showed the service as fixed.

[dersteppenwolf]

I added a new parameter GHC_VERIFY_SSL to enable / disable SSL verification for python request calls (get / post) https://github.com/dersteppenwolf/GeoHealthCheck/blob/master/GeoHealthCheck/probe.py#L290

These are the new commits:

https://github.com/dersteppenwolf/GeoHealthCheck/commit/1d5a086fc380a09304f70c7392c5179a923e3e4b

https://github.com/dersteppenwolf/GeoHealthCheck/commit/f2979bef4d8b7dbe4790535bd02a6fdccdfcc7c8

[borrob]

This looks like a nice solution to me (although I would keep the default to GHC_VERIFY_SSL = True). Can we add this to the main repository?

[dersteppenwolf]

sure, no problem

[borrob]

Giving it a little more thought: instead of True or False, it's also possible to verify with a separate certificate (eg. self-assigned certificate that we trust). See the documentation of requests.

This means we can/should keep track of a (path to) SSL certificate per resource, and thus add an extra column to the resource model. The certificates themselves should be supplied in a directory, similar to a directory with user defined plugins. In the front end the user should then be able to pick a supplied certificate, no verification, or use the standard certificates (based on certifici). How does that sound?

[dersteppenwolf]

It sounds like the right way to do it for those organizations that are too strict about security.

[chrismayer]

I have a similar problem. I configured a failure notification in GHC against an SSL secured endpoint. Then I get the following error message:

Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.site.de', port=443): Max retries exceeded with url: /hooks/abcdm5ose1234567bqw8c (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

So ignoring the SSL verification would be OK for us as an intermediate solution. In my docker-compose.yml I configured PYTHONHTTPSVERIFY=0 (as stated above), so it looks like that (excerpt):

  ghc_runner:
    image: geopython/geohealthcheck:latest

    container_name: ghc_runner

    restart: unless-stopped

    environment:
      - ADMIN_NAME=foo
      - ADMIN_PWD=bar
      - GHC_NOTIFICATIONS=True
      - PYTHONHTTPSVERIFY=0

Unfortunately this did not work. Does anyone got skipping SSL verification to work by ENV VARs? The approach of changing the source code is not suitable for us at the moment. Thanks in advance for any hint.

[borrob]

@chrismayer which code base/docker build did you use? I don't think this feature is added yet to the main repository.

[justb4]

@chrismayer @borrob is right. PYTHONHTTPSVERIFY=0 is a Python global variable, not an OS-env/GHC var. So to use this, a new GHC config var like GHC_VERIFY_SSL=True|False is needed that can propagate into PYTHONHTTPSVERIFY=0 as @dersteppenwolf indicated above. There is no PR yet.

[chrismayer]

@borrob I am using geopython/geohealthcheck:latest. Sorry forgot to mention that.

@justb4 Thanks for clarification. Now it becomes clearer to me (sorry, I am no python guy :wink: )

But what makes me wondering is that the probe itself against a server with a similar SSL certificate works, whereas the notification does not work due to SSL: CERTIFICATE_VERIFY_FAILED. Both servers have a letsencrypt certificate. Is there a different technology used within the probes than used within the notification? Thanks for your help!

[justb4]

Solved at least for Probe-originated HTTPS requests in PR #427 by @Maretius. Though some Probes use OWSLib that may still be a problem. So keeping this issue open.

[tobias-brunner]

To disable verification for all requests, you can patch Python37\Lib\site-packages\requests\adapters.py:

in def cert_verify(self, conn, url, verify, cert) set if url.lower().startswith("https") and verify: to always return false (e.g. if False:)

But: This is a last resort kind of solution, as now all requests made from this python-installation are unsafe!