Unable to verify SSL certificate when trying to build `cp311-macosx_x86_64`

[andyfaff]

Description

I'm trying to build a wheel for cp311-macosx_x86_64. Part of the CIBW_BEFORE_BUILD retrieves a pre-built OpenBLAS library using the urllib module.

We're experiencing issues retrieving that library with the Python install that cibuildwheel provides for that target, SSL verification is failing. We don't have the issue on any other target.

++ python tools/openblas_support.py
  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1348, in do_open
      h.request(req.get_method(), req.selector, req.data, headers,
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
      self._send_request(method, url, body, headers, encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
      self.endheaders(body, encode_chunked=encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
      self._send_output(message_body, encode_chunked=encode_chunked)
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
      self.send(msg)
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
      self.connect()
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line [145](https://github.com/scipy/scipy/pull/17046/checks#step:6:151)4, in connect
      self.sock = self._context.wrap_socket(self.sock,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
      return self.sslsocket_class._create(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1075, in _create
      self.do_handshake()
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1346, in do_handshake
      self._sslobj.do_handshake()
  ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/runner/work/scipy/scipy/tools/openblas_support.py", line 345, in <module>
      print(setup_openblas())
            ^^^^^^^^^^^^^^^^
    File "/Users/runner/work/scipy/scipy/tools/openblas_support.py", line 132, in setup_openblas
      typ = download_openblas(tmp, plat, ilp64)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/runner/work/scipy/scipy/tools/openblas_support.py", line 100, in download_openblas
      response = urlopen(req)
                 ^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 216, in urlopen
      return opener.open(url, data, timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
      response = self._open(req, data)
                 ^^^^^^^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
      result = self._call_chain(self.handle_open, protocol, protocol +
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
      result = func(*args)
               ^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1391, in https_open
      return self.do_open(http.client.HTTPSConnection, req,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1351, in do_open
      raise URLError(err)
  urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>

Build log

https://github.com/scipy/scipy/actions/runs/3083178264/jobs/4983796179

CI config

https://github.com/scipy/scipy/blob/main/.github/workflows/wheels.yml

[joerick]

Hm. Interesting. We do test SSL support for each version in our test suite, but this might be an unusual case. Could you post the URL that it's trying to download?

[andyfaff]

I think the URL was https://anaconda.org/multibuild-wheels-staging/openblas-libs/v0.3.18/download/openblas-v0.3.18-macosx_11_0_arm64-gf_f26990f.tar.gz

[joerick]

That URL uses a cloudflare-issued cert.

$  curl -v -X HEAD https://anaconda.org/multibuild-wheels-staging/openblas-libs

--snip--

* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=anaconda.org
*  start date: May  5 00:00:00 2022 GMT
*  expire date: May  5 23:59:59 2023 GMT
*  subjectAltName: host "anaconda.org" matched cert's "anaconda.org"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing

I can't seem to test that in Python though, I suppose the CDN isn't letting it through.

>>> urllib.request.urlopen('https://anaconda.org/multibuild-wheels-staging/openblas-libs/v0.3.18/download/openblas-v0.3.18-macosx_11_0_arm64-gf_f26990f.tar.gz') 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Other cloudflare-hosted sites like https://cloudflare.com use TLSv1.3 and a different cipher, but I guess the cert verification is the same? I'll try adding https://cloudflare.com to our test suite, lets see what happens.

[joerick]

Not getting anywhere with this... I can't find a Cloudflare URL works, every one that I've tried responds to urllib.request.urlopen with a 403 Forbidden error - I suppose we're being treated as a bot. I don't much want to fiddle with user-agent etc as we'd be fighting cloudflare at their own game.

I can tell you that even on Python 3.11, we do run pip install --upgrade certifi, so the certificates should be up-to-date.

[andyfaff]

The download script we use specifies user agents, etc https://github.com/scipy/scipy/blob/main/tools/openblas_support.py

[joerick]

Oh, that's been working reliably for you? I'll try it.

[andyfaff]

It's called as python open_blas.py

[joerick]

I found the issue. #1279 shows this is not normally an issue in cibuildwheel. Your Github action requests setup-python to install 3.11-dev. That GHA install script uses a Python.org pkg, but neglects to install SSL certificates from certifi. And since 3.11 is already installed on the build system, cibuildwheel doesn't reinstall it, and doesn't fiddle with SSL certs.

In your case, a quick fix would be

/Library/Frameworks/Python.framework/Versions/3.11/bin/python -m pip install certifi

However, it's probably worth sending a issue/PR to actions/setup-python since I'd consider this an incomplete install of Python, and this will affect many others too.

[henryiii]

(An alternative fix would be what I suggested earlier in that PR, to not install python first, as it's unneeded for the cibuildwheel action, and in this case actually breaks it!)

[mayeut]

An alternative fix would be what I suggested earlier in that PR, to not install python first, as it's unneeded for the cibuildwheel action, and in this case actually breaks it!

Well, this would be a short-term fix/workaround, as @joerick mentioned, setup-python is moving to using Python.org pkg so we'll get a pre-installed 3.11 next month that uses the same script as setup-python on the runners and it'll be a broken install.

it's probably worth sending a issue/PR to actions/setup-python since I'd consider this an incomplete install of Python, and this will affect many others too.

I'll do that over the week-end.

[amotl]

Hi there,

we probably just observed the same issue on https://github.com/crate/crate-python/pull/450, so we submitted a report at https://github.com/actions/setup-python/issues/512. In the meanwhile, we are using https://github.com/crate/crate-python/commit/243820d1 as a workaround.

With kind regards, Andreas.

[mayeut]

I opened https://github.com/actions/python-versions/pull/189 to fix this issue.

[amotl]

Hi again,

your fix works like a charm, @mayeut. Thank you very much for submitting it so quickly.

With kind regards, Andreas.

[mayeut]

This ended up being an issue with actions/setup-python: https://github.com/actions/setup-python/issues/512 - now fixed.

[tylerjereddy]

This is still pretty painful for me locally--I have quite a mix of brew-based venv and standard CPython/Python.org installs available, and the homebrew-based venv can do python tools/openblas_support.py just fine, but a venv created from the python.org 3.11 install encouters the exact same traceback from the original comment above, inside or outside of cibuildwheel.

I suppose this doesn't have much to do with cibuildwheel, apart from the need to use the Python.org Python.

[tylerjereddy]

For the record, no pip install on its own helped, I had to do this as root on my mac:

• cd /Applications/Python\ 3.11
• ./Install\ Certificates.command