contextily
contextily copied to clipboard
geopandas
third-party SSL certificate?
For those of us working behind stiff security measures is there a way to pass in the path to a third-party SSL certificate? I can install and import contextily fine, but as soon as I try to load tiles I inevitably get a SSLCertVerificationError.
Thank you!
I suppose this error is coming from requests, which we use to download the tiles ? (maybe https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification)
I am not very familiar with this, but I suppose that requests has some interface to specify third-party SSL certificates? If that is the case, we can certainly try to provide an interface to pass through options for requests, I think.
Yup, if you just provide an argument that gets passed to requests.get() or .post(), that should do it. I've only used the verify kwarg, but an alternative is cert. That would provide the most flexibility for those who either want to pass a third-party cert or cowboy it with verify=False.
has there been any further work in this area? A coworker just ran into this issue. I can't replicate it on my end, unfortunately. (his network is locked down tighter than mine).
has there been any further work in this area? A coworker just ran into this issue. I can't replicate it on my end, unfortunately. (his network is locked down tighter than mine).
I haven't tried to write something, but I've been meaning to... should be pretty straightforward.
Chiming in here as someone who has those aforementioned stiff security measures - it would be great to be able to pass the verify kwarg (or something similar) to add_basemap() (to be passed torequests.get()):
https://github.com/geopandas/contextily/blob/main/contextily/tile.py#L395
Here's the full traceback from the SSL cert error:
---------------------------------------------------------------------------
SSLCertVerificationError Traceback (most recent call last)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connectionpool.py:700, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
699 if is_new_proxy_conn and http_tunnel_required:
--> 700 self._prepare_proxy(conn)
702 # Make the request on the httplib connection object.
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connectionpool.py:996, in HTTPSConnectionPool._prepare_proxy(self, conn)
994 conn.tls_in_tls_required = True
--> 996 conn.connect()
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connection.py:414, in HTTPSConnection.connect(self)
412 context.load_default_certs()
--> 414 self.sock = ssl_wrap_socket(
415 sock=conn,
416 keyfile=self.key_file,
417 certfile=self.cert_file,
418 key_password=self.key_password,
419 ca_certs=self.ca_certs,
420 ca_cert_dir=self.ca_cert_dir,
421 ca_cert_data=self.ca_cert_data,
422 server_hostname=server_hostname,
423 ssl_context=context,
424 tls_in_tls=tls_in_tls,
425 )
427 # If we're using all defaults and the connection
428 # is TLSv1 or TLSv1.1 we throw a DeprecationWarning
429 # for the host.
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/util/ssl_.py:449, in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
448 if send_sni:
--> 449 ssl_sock = _ssl_wrap_socket_impl(
450 sock, context, tls_in_tls, server_hostname=server_hostname
451 )
452 else:
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/util/ssl_.py:493, in _ssl_wrap_socket_impl(sock, ssl_context, tls_in_tls, server_hostname)
492 if server_hostname:
--> 493 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
494 else:
File /srv/conda/envs/notebook/lib/python3.9/ssl.py:501, in SSLContext.wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
495 def wrap_socket(self, sock, server_side=False,
496 do_handshake_on_connect=True,
497 suppress_ragged_eofs=True,
498 server_hostname=None, session=None):
499 # SSLSocket class handles server_hostname encoding before it calls
500 # ctx._wrap_socket()
--> 501 return self.sslsocket_class._create(
502 sock=sock,
503 server_side=server_side,
504 do_handshake_on_connect=do_handshake_on_connect,
505 suppress_ragged_eofs=suppress_ragged_eofs,
506 server_hostname=server_hostname,
507 context=self,
508 session=session
509 )
File /srv/conda/envs/notebook/lib/python3.9/ssl.py:1041, in SSLSocket._create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
1040 raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1041 self.do_handshake()
1042 except (OSError, ValueError):
File /srv/conda/envs/notebook/lib/python3.9/ssl.py:1310, in SSLSocket.do_handshake(self, block)
1309 self.settimeout(None)
-> 1310 self._sslobj.do_handshake()
1311 finally:
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/adapters.py:489, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
488 if not chunked:
--> 489 resp = conn.urlopen(
490 method=request.method,
491 url=url,
492 body=request.body,
493 headers=request.headers,
494 redirect=False,
495 assert_same_host=False,
496 preload_content=False,
497 decode_content=False,
498 retries=self.max_retries,
499 timeout=timeout,
500 )
502 # Send the request.
503 else:
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connectionpool.py:787, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
785 e = ProtocolError("Connection aborted.", e)
--> 787 retries = retries.increment(
788 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
789 )
790 retries.sleep()
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/util/retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
591 if new_retry.is_exhausted():
--> 592 raise MaxRetryError(_pool, url, error or ResponseError(cause))
594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
MaxRetryError: HTTPSConnectionPool(host='stamen-tiles-a.a.ssl.fastly.net', port=443): Max retries exceeded with url: /terrain/3/0/1.png (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))
During handling of the above exception, another exception occurred:
SSLError Traceback (most recent call last)
Cell In[27], line 2
1 ax = all_endorheic_basins.plot(figsize=(20, 20))
----> 2 ctx.add_basemap(ax, crs="EPSG:4327")
File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/plotting.py:121, in add_basemap(ax, zoom, source, interpolation, attribution, attribution_size, reset_extent, crs, resampling, **extra_imshow_args)
117 left, right, bottom, top = _reproj_bb(
118 left, right, bottom, top, crs, {"init": "epsg:3857"}
119 )
120 # Download image
--> 121 image, extent = bounds2img(
122 left, bottom, right, top, zoom=zoom, source=source, ll=False
123 )
124 # Warping
125 if crs is not None:
File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/tile.py:222, in bounds2img(w, s, e, n, zoom, source, ll, wait, max_retries)
220 x, y, z = t.x, t.y, t.z
221 tile_url = provider.build_url(x=x, y=y, z=z)
--> 222 image = _fetch_tile(tile_url, wait, max_retries)
223 tiles.append(t)
224 arrays.append(image)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/joblib/memory.py:594, in MemorizedFunc.__call__(self, *args, **kwargs)
593 def __call__(self, *args, **kwargs):
--> 594 return self._cached_call(args, kwargs)[0]
File /srv/conda/envs/notebook/lib/python3.9/site-packages/joblib/memory.py:537, in MemorizedFunc._cached_call(self, args, kwargs, shelving)
534 must_call = True
536 if must_call:
--> 537 out, metadata = self.call(*args, **kwargs)
538 if self.mmap_mode is not None:
539 # Memmap the output at the first call to be consistent with
540 # later calls
541 if self._verbose:
File /srv/conda/envs/notebook/lib/python3.9/site-packages/joblib/memory.py:779, in MemorizedFunc.call(self, *args, **kwargs)
777 if self._verbose > 0:
778 print(format_call(self.func, args, kwargs))
--> 779 output = self.func(*args, **kwargs)
780 self.store_backend.dump_item(
781 [func_id, args_id], output, verbose=self._verbose)
783 duration = time.time() - start_time
File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/tile.py:252, in _fetch_tile(tile_url, wait, max_retries)
250 @memory.cache
251 def _fetch_tile(tile_url, wait, max_retries):
--> 252 request = _retryer(tile_url, wait, max_retries)
253 with io.BytesIO(request.content) as image_stream:
254 image = Image.open(image_stream).convert("RGBA")
File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/tile.py:395, in _retryer(tile_url, wait, max_retries)
375 """
376 Retry a url many times in attempt to get a tile
377
(...)
392 request object containing the web response.
393 """
394 try:
--> 395 request = requests.get(tile_url, headers={"user-agent": USER_AGENT})
396 request.raise_for_status()
397 except requests.HTTPError:
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs)
62 def get(url, params=None, **kwargs):
63 r"""Sends a GET request.
64
65 :param url: URL for the new :class:`Request` object.
(...)
70 :rtype: requests.Response
71 """
---> 73 return request("get", url, params=params, **kwargs)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs)
55 # By using the 'with' statement we are sure the session is closed, thus we
56 # avoid leaving sockets open which can trigger a ResourceWarning in some
57 # cases, and look like a memory leak in others.
58 with sessions.Session() as session:
---> 59 return session.request(method=method, url=url, **kwargs)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/sessions.py:587, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
582 send_kwargs = {
583 "timeout": timeout,
584 "allow_redirects": allow_redirects,
585 }
586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
589 return resp
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/sessions.py:701, in Session.send(self, request, **kwargs)
698 start = preferred_clock()
700 # Send the request
--> 701 r = adapter.send(request, **kwargs)
703 # Total elapsed time of the request (approximately)
704 elapsed = preferred_clock() - start
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/adapters.py:563, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
559 raise ProxyError(e, request=request)
561 if isinstance(e.reason, _SSLError):
562 # This branch is for urllib3 v1.22 and later.
--> 563 raise SSLError(e, request=request)
565 raise ConnectionError(e, request=request)
567 except ClosedPoolError as e:
SSLError: HTTPSConnectionPool(host='stamen-tiles-a.a.ssl.fastly.net', port=443): Max retries exceeded with url: /terrain/3/0/1.png (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))
One potential workaround is to use os.environ['REQUESTS_CA_BUNDLE'] = your_cert_path somewhere in your own script before adding tiles with contextily. This sets the certificate to use globally while your script is running (you could also presumably set it for your OS, but then your script is less portable). I haven't been able to reproduce the contextily-specific requests issue for some reason, but it does resolve my problem when calling requests.get() directly.
I can confirm that the above workaround using os.environ resolves my original issue.