SSL is broken on python 3.11 on M1 mac running Monterey

[pragbatra-mendel]

Despite installing python 3.11 pkg from the python website and running the Install Certificates script which completes without error, I still get ssl certificate verify failed whenever I try to connect to a website that uses SSL. Running without verifying SSL certificates works fine. When I run openssl version I get LibreSSL 2.8.3, even though I installed openssl via homebrew. Is there any way to fix this?

[ned-deily]

Are you sure that you are really using the python installed from the python.org website? It is very easy to be confused if you have more than one version of python installed; for instance, Apple provides a python3 with its Xcode / Command Line Tools. The easiest way to verify which you are using it to print the system version information along with a simple failing test; for the python.org 3.11.0 release, it should show v3.11.0:deaf509e8f, Oct 24 2022, 14:43:23. And other versions of OpenSSL, like provided by Homebrew, have no effect on the python.org-installed Python on macOS.

[pragbatra-mendel]

Yes, I am using the official python installer and when I run python --version it shows 3.11.0. I did brew install openssl and brew link --force openssl and openssl version now shows OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) instead of LibreSSL. When I try to download a file from an https site with urllib.request.urlopen().read().decode() it gives the below error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1454, in connect
    self.sock = self._context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>

[ned-deily]

Without more information, it is difficult to explain what you are reporting. If you are indeed using the python.org installation, it has its own embedded version of OpenSSL 1.1.1; it does not use or is affected at all by an OpenSSL installed via Homebrew or any other mechanism.

Please execute the following two commands in your terminal shell and paste the output here:

/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11 -c 'import sys;print(sys.version)'
/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11 -m pip list

Please also copy and paste the exact sequence of commands including the URL that gives the CERTIFICATE_VERIFY_FAILED error.

[pragbatra-mendel]

/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11 -m pip list
3.11.0 (v3.11.0:deaf509e8f, Oct 24 2022, 14:43:23) [Clang 13.0.0 (clang-1300.0.29.30)]
Package            Version
------------------ ----------
certifi            2022.9.24
cffi               1.15.1
charset-normalizer 2.1.1
cryptography       38.0.3
idna               3.4
pip                22.3
pycparser          2.21
pyOpenSSL          22.1.0
requests           2.28.1
setuptools         65.5.0
urllib3            1.26.12

In my current environment, it looks like the error is specific to the URL of the file I'm trying to download. I'll check if it's reproducible on a working configuration on Intel architecture to see if the issue is specific to M1. Curl downloads the file fine so I suspect it is still an issue with python certificates however.

[ned-deily]

Thanks for the response, that does look correct for the python.org 3.11.0 release. Whichever version of Curl you are using, it is undoubtedly using a somewhat different set of root certificates (i.e. it is not using those provided by the PyPI certifi package which is what is installed by Install Certificates.command so that might be an issue. You could try to check the certificate chain using OpenSSL and/or curl; see, for example, this.

[ronaldoussoren]

When I try to download a file from an https site with urllib.request.urlopen().read().decode() it gives the below error:

Does this happen with all websites or just some of them? Is one of the problematic websites a public website that you can share the name for (so that we can look at the certificate ourselves)?

[ronaldoussoren]

I propose closing this issue: The issue is likely in a particular website that has a certificate with a chain that's not rooted at a certificate trusted by the CA roots included with certifi. That's not something we can fix on our end.

[ned-deily]

I agree.