indygreg
Python unable to see CA certificates in Fedora/Rocky/Alma/RHEL systems
Hi,
I think there is an issue when attempting to use this precompiled binaries on a rhel-based system. I've read the quirks section, and I haven't seen a reference to this problem, that is why I'm raising an issue.
The reason we care about this funcionality is because we're trying to use this python + poetry to get "portable" dev environments accross a bunch of developers (using mise).
Issue description
The following code reproduces the problem on a rhel-based system:
docker run -it --rm rockylinux:8 bash -c \
"dnf update -y && dnf install -y wget ca-certificates && \
wget https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
tar -xzf cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
./python/bin/python -c \"import urllib.request; f = urllib.request.urlopen('https://www.python.org'); print(f.read(100))\""
Logs
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27409290 (26M) [application/octet-stream]
Saving to: 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz'
cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-in 100%[=====================================================================================================================>] 26.14M 8.59MB/s in 3.0s
2024-05-06 08:57:30 (8.59 MB/s) - 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz' saved [27409290/27409290]
Traceback (most recent call last):
File "/python/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/python/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/python/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/python/lib/python3.10/http/client.py", line 1455, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/python/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/python/lib/python3.10/ssl.py", line 1104, in _create
self.do_handshake()
File "/python/lib/python3.10/ssl.py", line 1375, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/python/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/python/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/python/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/python/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/python/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/python/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>
But the same python program, urlopen works just fine on a debian-based system.
docker run -it --rm debian:12 bash -c \
"apt update -y && apt install -y wget ca-certificates && \
wget https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
tar -xzf cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
./python/bin/python -c \"import urllib.request; f = urllib.request.urlopen('https://www.python.org'); print(f.read(100))\""
Logs
...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27409290 (26M) [application/octet-stream]
Saving to: 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz'
cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-in 100%[=====================================================================================================================>] 26.14M 8.63MB/s in 3.0s
2024-05-06 08:53:54 (8.63 MB/s) - 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz' saved [27409290/27409290]
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xed}\xebr\x1bG\xb2\xe6\x7fE\xcc;\x94\xa1X\x91\x1a\xab\x01\x82\xe0M\x14\t\x8f$S2=\xba\xd0\xa6d\x9d\xb3>\x0eE\xa3\xd1\x00\x9a\x04\xba\xa1\xbe\x90\x84g\xe6\x01\xce\x9f}\x85}\xc5}\x84\xfd2\xab\xaa\xbb\xfa\n@\xe4\xe8\xd8\x11\x9a\t\x8b@\xa3\xba.Yy\xaf\xcc\xac\xa3o\x86\x81\x13/\xe6'
Other things I've tried
I've tried installing pip-system-certs and certifi, using the provided pip, but neither of them fixed the issue.
Thank you for reading this far, and let me know if there is something else I can do to help. I'm stuck and unable to fix this by myself (I think).
I ran into the exact same thing on Python 3.10 (3.8 worked fine) , and used monkey patching in the sitecustomize.py to enable urllib ssl globally per ChatGPT recommendation:
vi $(python -c "import site; print(site.getsitepackages()[0])")/sitecustomize.py
paste this
import ssl
try:
import certifi
# Define a function that returns a default SSL context with certifi's CA bundle
def create_certifi_context(purpose=ssl.Purpose.SERVER_AUTH, *, cafile=None, capath=None, cadata=None):
return ssl.create_default_context(purpose, cafile=certifi.where(), capath=capath, cadata=cadata)
# Set the default SSL context creation function to use certifi's CA bundle
ssl._create_default_https_context = create_certifi_context
except:
print('certifi package not installed')
I also ran into this. Explicitly specifying the SSL certificate file seems to fix the issue:
export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
Is this a problem of the standalone builds? i.e., is this reproducible with a local build with pyenv? I wonder if this is just a difference in how the certificates are configured on the systems?
Just built CPython 3.11.9 with pyenv and tested but this issue didn't happen.
After some digging, I've found that RHEL-based distros store their root CA certificate bundle as /etc/ssl/certs/ca-bundle.crt while Debian-based distros store it as/etc/ssl/certs/ca-certificates.crt. Also these paths seem to be baked in when OpenSSL is built. So yes, this comes from the different CA bundle paths between distros.