Feature request: Pass certificates and keys without writing them to disk

[whiteowl3]

Initially raised as discussion #2037 Relevant discussion in #924

PyOpenSSL gives us the ability to create ssl contexts without writing keys or certificates to memory by calling the existing (foreign C) openssl functions:

https://github.com/pyca/pyopenssl/blob/2bd1c5104a7a55715db93157e4f6a657a4f2a154/src/OpenSSL/SSL.py#L952 https://github.com/pyca/pyopenssl/blob/2bd1c5104a7a55715db93157e4f6a657a4f2a154/src/OpenSSL/_util.py#L10 https://github.com/pyca/cryptography/blob/1cc4a6e786b97d3cfa2899d4ddb3a3fabc2abd12/src/_cffi_src/openssl/ssl.py#L215 https://cryptography.io/en/3.1/hazmat/bindings/openssl/

An example of using this functionality (sync, blocking) can be found here: https://github.com/encode/httpx/issues/924#issuecomment-1052681712

httpx should implement this functionality in its sync and async clients, either by monkey-patching PyOpenSSL contexts a la urllib3, or, more fun, allowing the client certificate argument to accept key/cert pairs of all relevant types and use the ffi lib directly where necessary, such that the argument passed to cert could be any combination of cryptography's certificate and key objects, pyopenssl's PKey and X509 objects, and httpx-style string/tup arguments. The ideal solution (imho) would allow arbitrary combinations of any of these (a file path to a pem key paired with a pyopenssl X509 object, for example, or a cryptography x509 cert paired with a pyopenssl PKey).

[whiteowl3]

the urllib3 monkey patch: https://github.com/urllib3/urllib3/blob/9f4d05c11eb2f80682c1b7a2671b0636292ef932/src/urllib3/contrib/pyopenssl.py#L149

[tomchristie]

Here's where I got to with attempting to use pyopenssl directly with httpcore...

import httpcore
from OpenSSL import SSL


class SSLContext(SSL.Context):
    def set_alpn_protocols(self, protos):
        # The stdlib uses `set_alpn_protocols`, which is named slightly
        # different from the `pyopenssl` version. Also the `pyopenssl`
        # expects the arguments a list of bytes, not a list of str.
        super().set_alpn_protos([p.encode("ascii") for p in protos])

    def wrap_socket(self, sock, server_hostname=None):
        # The `pyopenssl` Context doesn't provide a `wrap_socket` method.
        #
        # Note: I'm unclear how to set the `server_hostname` here.
        conn = SSL.Connection(self, sock)
        conn.set_connect_state()
        conn.do_handshake()
        return conn


pool = httpcore.ConnectionPool(ssl_context=SSLContext(SSL.SSLv23_METHOD))
r = pool.request("GET", "https://www.example.com")
print(r)

This is a slightly neater starting point, since httpcore.ConnectionPool accepts an ssl_context argument directly.

(By contrast, httpx accepts verify, cert, and trust_env arguments to setup the ssl_context argument, and there's a little extra hoop-jumping you'd need to do to pass a custom SSLContext all the way through.)

A good starting point for this issue would be for someone to check if the example I've provided above is complete, and works as expected or not.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[whiteowl3]

Not stale.

[deepsurfacesec]

Another reason to allow usage of pyopenssl is that the ssl module doesn't have any way to control the verification callback. Specifically, I'm referring to this method: https://www.pyopenssl.org/en/latest/api/ssl.html#OpenSSL.SSL.Context.set_verify

If you have a need, as I do, to do more in-depth inspection of server certificates during a handshake, this allows you to define a callback function that can perform those checks. The built-in ssl module has no way to do this, which is super disappointing.

With requests and urllib3, they used to have the ability to use a pyOpenSSL backend. Then with a simple adapter you could set the verification callback. But now they're deprecating pyOpenSSL and it is really hard to find any HTTP library whose authors care about pyOpenSSL and any advanced certificate management. :-(

[deepsurfacesec]

Here's what I was able to get working, based on @tomchristie's post a year ago:

ctx = certificates.TOFUContext(verifier, 'any_tls', logger=logger)

pool = httpcore.ConnectionPool(ssl_context=ctx)
transport = httpx.HTTPTransport()
transport._pool = pool
client = httpx.Client(transport=transport, timeout=None)
r = client.request("GET", f"https://{hostname}")
print(r.text)

The TOFUContext is a custom context very similar to what Tom posted, but is tied in to some other logic that leverages the set_verify function and callback that I mentioned earlier.

The two parts that make this a little hacky are:

• The need to override the _pool attribute after initializing the HTTPTransport (subclassing wasn't convenient given how complex the constructor is)
• The need to set timeout=None. This was needed because the SyncStream.read() method was blowing up for me when the pyOpenSSL-based socket was set to non-blocking. It raises WantReadError when there's no data available to read... I guess? I don't fully understand that bit yet, but I've run into it before. I'm currently on pyopenssl 19.0.0

[post-j-ma]

Nice to see a solution that doesn't depend on urllib3's support for pyOpenSSL but goes directly (since as already noted above, that's ending, https://github.com/urllib3/urllib3/issues/2680).

[deepsurfacesec]

Unfortunately my solution above still has race conditions. Even though I'm setting timeout=None, somewhere along the line the socket is switched to nonblocking and that results in race conditions during the SSL/TLS handshake. I'm hoping to get to the bottom of that soon.

[tomchristie]

I'm going to close this off now since it's not something that we're going to handle.

Python's SSLContext doesn't support this... see https://github.com/python/cpython/issues/60691 and https://github.com/python/cpython/pull/2449

There's enough information here on the pyopenssl workaround for someone to be able to work with. If anyone digs into that feel free to update this thread for the sake of others, or consider contributing to the third party packages docs. Will happily help work together with folks to help make any workarounds there more widely available but we're not currently able to address this directly.

Also discussed here... https://github.com/encode/httpx/discussions/2037