obtain SSL certificates with Python >= 3.6 on macOS

[johnyf]

Running python setup.py install --fetch --cudd with Python 3.6 (built from source) on OS X can encounter the following error due to absent SSL certificates:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)>

Solution:

• Install SSL certificates for Python: https://stackoverflow.com/a/42334357/1959808

Other approaches (with reference to dc0b3c3a178064a3eeb1d2a8245bc6635c52860f):

• pip install certifi, as recommended by CPython (see also Python 3.6 release notes and Python Issue29065.

diff --git a/download.py b/download.py
index 81a228c..711e279 100644
--- a/download.py
+++ b/download.py
@@ -12,6 +12,7 @@ except ImportError:
     import urllib.request, urllib.error, urllib.parse
     urllib2 = urllib.request

+import certifi
 try:
     import cysignals
 except ImportError:
@@ -152,7 +153,7 @@ def _join(paths):

 def fetch(url, sha256, fname=None):
     print('++ download: {url}'.format(url=url))
-    u = urllib2.urlopen(url)
+    u = urllib2.urlopen(url, cafile=certifi.where())
     if fname is None:
         fname = CUDD_TARBALL
     with open(fname, 'wb') as f:

• Manually download cudd-3.0.0.tar.gz to the setup.py directory (e.g., from SourceForge), unpack, and run make with the appropriate flags. The flags are listed inside download.py, so a shortcut after unpacking is:

  python -c 'from download import make_cudd; make_cudd()'
  python setup.py install --cudd
  

• The tarball hash is checked anyway, so bypassing SSL certificate verification inside download.py is an (unrecommended) option.

diff --git a/download.py b/download.py
index 81a228c..b89b2d4 100644
--- a/download.py
+++ b/download.py
@@ -3,6 +3,7 @@ import ctypes
 import hashlib
 import os
 import shutil
+import ssl
 import subprocess
 import sys
 import tarfile
@@ -152,7 +153,8 @@ def _join(paths):

 def fetch(url, sha256, fname=None):
     print('++ download: {url}'.format(url=url))
-    u = urllib2.urlopen(url)
+    context = ssl._create_unverified_context()
+    u = urllib2.urlopen(url, context=context)
     if fname is None:
         fname = CUDD_TARBALL
     with open(fname, 'wb') as f:

[johnyf]

The post-install script to install SSL certificates for Python 3.6 (mentioned here) first installs the certifi package, and then links the certificates file of OpenSSL to that of the certifi package. The script is at:

https://github.com/python/cpython/blob/cebe9ee988837b292f2c571e194ed11e7cd4abbb/Mac/BuildScript/resources/install_certificates.command

Therefore, this issue is about installing Python 3.6 with SSL access to SSL certificates. The script download.py works without changes after the above post-install script is run with Python 3.6. So introducing an additional package as dependency of the script download.py is not needed.

[johnyf]

Building Python from source on macOS

It appears that:

• having an installation of OpenSSL on macOS when building Python, and
• having or installing SSL certificates

are needed on macOS since Python 3.6.

When building CPython >= 3.7 from source, the script configure has an option --with-openssl to override the root of the OpenSSL directory: https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/configure#L1634

The following documentation can be useful: https://github.com/python/devguide/blob/b9f1ae8c60c62d1e709659f016e73b94e0cd4d41/setup.rst#macos-and-os-x quoting from there:

As of OS X 10.11, Apple no longer provides header files for the deprecated system version of OpenSSL which means that you will not be able to build the _ssl extension. One solution is to install these libraries from a third-party package manager, like Homebrew or MacPorts, and then add the appropriate paths for the header and library files to your configure command.

(See also this post.)

Looking at my Python installations, it seems that the following are the cases:

• Python 3.7, 3.8, 3.9 use the MacPorts openssl, and its certificates file cert.pem is a symbolic link to the file .../curl-ca-bundle.crt of MacPorts. In fact, as revealed by port contents curl-ca-bundle, these two files are both installed by the MacPorts subport curl-ca-bundle: https://github.com/macports/macports-ports/blob/9041c30899bd58eeba9959f0f8cc952d47136bf7/net/curl/Portfile#L285 Looking in the file curl-ca-bundle.crt, it reads

  ## Bundle of CA Root Certificates ## ## Certificate data from Mozilla as of: ...

  This provenance of cURL's certificate bundle is confirmed also from cuRL's website (as mentioned there, the script mk-ca-bundle.pl creates cURL's certificate bundle). The URL mentioned in the certificates file is: https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt These observations suggest that ssl on these Python installations uses the Mozilla certificates bundle.

• Python 3.6 uses an openssl that appears to have been installed from source by me, and its certificates file cert.pem is a symbolic link to the file .../site-packages/certifi/cacert.pem. Based on what the Python package certifi describes in its file README.rst, and the repository cert-tools, the certificates bundle of certifi is created from the Mozilla certificates bundle using the the tool in the repository extract-nss-root-certs. The URL mentioned in the file README.markdown is: https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt

So both the cURL certificates bundle and the certifi certificates bundle appear to be derived from the Mozilla certificates bundle.

The relevant commands for finding this information are:

import pprint
import ssl
import sysconfig

print(ssl.get_default_verify_paths())
pprint.pprint(sysconfig.get_config_vars())  # look for "OPENSSL" in the results

I installed Python 3.10 from source, and without passing any information about OpenSSL to the script configure, the script decided to use the OpenSSL that has been installed by MacPorts. It is not obvious why configure chose the OpenSSL installed by MacPorts. From looking at the source code of configure, it appears that it looks for pkg-config (which on my macOS is installed by only MacPorts, namely as the port pkgconfig), and it asks pkg-config about OpenSSL-related paths (relevant area in configure appears to be this, more specifically this comment is informative, together with these lines). The values returned by pkg-config openssl --libs-only-L, pkg-config openssl --cflags-only-I, and pkg-config openssl --libs-only-l on my macOS indeed match the values of the keys 'OPENSSL_LDFLAGS', 'OPENSSL_INCLUDES', and 'OPENSSL_LIBS', respectively, in the dict that is returned by:

python3 -c "import sysconfig; import pprint; pprint.pprint(sysconfig.get_config_vars())"

These observations appear to explain why the Python build picks up the MacPorts OpenSSL.

Installing Python using an installer for macOS

The above notes are for installing CPython from source on macOS. If CPython is installed on macOS using an installer, it appears that it bundles its own build of OpenSSL, and needs a separate post-installation step by the user that installs the Python package certifi, and links the OpenSSL certificates to those of certifi. This information is from:

• https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/ReadMe.rtf#L22-L34 ("This package includes its own private copy of OpenSSL 1.1.1. ... A sample command script is included ... to install a curated bundle of default root certificates from the third-party ... certifi package ... Double-click on Install Certificates to run it.")
• https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Conclusion.rtf#L15-L19
• https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Welcome.rtf#L23-L25
• https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/install_certificates.command
• https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/README.rst ("builds the following third-party libraries ... OpenSSL 1.1.1")
• https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/build-installer.py#L239-L246

This is what the comment above (https://github.com/tulip-control/dd/issues/35#issuecomment-589950204) is about.