Importing Fiona sets default CURL_CA_BUNDLE environment variable interfering with requests library

[hugocarr]

Expected behavior and actual behavior.

I expected my os.environ environment to remain unchanged when importing fiona. However, when I import fiona (via geopandas) it adds a default value for CURL_CA_BUNDLE for the entire process: https://github.com/Toblerity/Fiona/blob/d36e0c897c545e0e51fe759e540c85c117bf3fc1/fiona/_env.pyx#L60

This interferes with the requests library for other parts of code using private CAs

Steps to reproduce the problem.

>>> import os
>>> assert "CURL_CA_BUNDLE" not in os.environ
>>> import fiona
>>> assert "CURL_CA_BUNDLE" in os.environ 

This is specifically a problem for an mTLS application such as the following:

import requests

def query_https():
    s = requests.Session()
    s.verify = "/tmp/certs/rest_cacert.crt"
    s.cert = ("/tmp/certs/rest_certificate.pem", "/tmp/certs/rest.key")
    resp = s.get("https://my_domain.com/my_route")
    print(resp.text)

query_https()
import fiona

query_https()

Resulting in one successful call, but then the second:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)

This is because the verify parameter in requests is overridden by the default environ value above. See: https://github.com/psf/requests/blob/79f2ec3acc4e24fef6e6ce31ad7b1d4e2f77be31/requests/sessions.py#L765-L770

Operating system

Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

Fiona and GDAL version and provenance

Fiona

version 1.8.21 installed from PyPI using pip version 22.1

GDAL

wget --quiet https://zipline-build-resources.s3.us-west-2.amazonaws.com/src/gdal-3.2.1.tar.gz && \
    tar xzf gdal-3.2.1.tar.gz && cd gdal-3.2.1 && ./configure --with-python && \
    make -j$(nproc) && make install \

[jorisvandenbossche]

See also https://github.com/Toblerity/Fiona/issues/1091 (I suppose this is a duplicate issue). And it should be fixed in a future version by https://github.com/Toblerity/Fiona/pull/1095.

[mwtoews]

I can verify the os.environ behaviour with 1.8.21 from PyPI, and this appears to be resolved with the latest prerelease. If possible, try using: pip install --pre --upgrade fiona, which may hopefully get binary wheels for your platform (i.e. does not require GDAL to be pre-installed).

The included fix for the 1.9a1 prerelease uses GDAL_CURL_CA_BUNDLE and PROJ_CURL_CA_BUNDLE variable names in os.environ instead of CURL_CA_BUNDLE. Any further suggestions are much appreciated before the next release gets out.

[hugocarr]

Thanks for that pointer @mwtoews! We will look into using the pre-release. In general I think that it's fine for a library to consume an environment variable, but I was definitely surprised to see that an import added new environment variables. (Especially as this was a side effect of importing geopandas).

From a hygiene perspective it might be preferable to use a context manager to temporarily set the environment variable where you need it in the fiona code. For example, I do this a lot in unitests. Thoughts?

[sgillies]

@hugocarr yes, we use a context manager for GDAL config options and could figure out a way to add the environment variable to that one.