CDRIVER-4656: Driver re-initializes OpenSSL Context on every new socket

[Julia-Garland]

Summary

For OpenSSL versions >= 1.1.0, single-threaded clients and client pools own a single OpenSSL context that is shared among all connections a client makes over TLS.

The issue

The C driver creates a new OpenSSL context for every new connection that a client makes over TLS. Creating an OpenSSL context is a relatively expensive operation that involves parsing the root certificate store for the operating system and building an internal data structure representing those certificates.

This performance issue was discovered as part of PERF-4166. When making 10,000 clients, it was parsing certificates a total of 50,000 times, multiple times for each client. This was enough to make the test essentially not generate any traffic and keep all the cores busy just parsing certificate files repeatedly.

What changed?

SSL context sharing:

• In the original behavior, a new SSL_CTX was being created locally from a client’s ssl_opts in each call to mongoc_stream_tls_openssl_new(), even though the SSL configuration is unlikely to change between connections.
• Upon single-threaded client or client pool creation, an SSL_CTX is created based on the ssl_opts field and is owned by the topology scanner. In the pooled case, any clients popped from the client pool will inherit the pool’s topology and thus its SSL_CTX.
• The SSL_CTX is used for all connections the client makes over TLS. The context is not modified after it is created, so there is no danger in multiple clients/connections sharing it.

Updates to ssl_opts:

• Changing the ssl_opts of a single-threaded client or a client pool with mongoc_client_set_ssl_opts() and mongoc_client_pool_set_ssl_opts() respectively will result in the SSL_CTX owned by that client or pool to be re-initialized to a context matching the new opts.
• The old SSL_CTX stays in existence for any connections still using it and is automatically freed when all connections using it have been cleaned up. Note this relies on reference counting (with SSL_CTX_up_ref()) which was first introduced in OpenSSL 1.1.0.

Running performance benchmarks

A multi-threaded benchmark I used to test the performance improvement of this PR can be found in I used two benchmarks to test the performance improvement of this PR. Both can be found in src/libmongoc/tests.

• benchmark-tls-pooled checks out 200 clients from a client pool configured with the driver's built-in TLS files, each of which sends a single ping command. The number of clients can be modified by an optional integer argument. The maxPoolSize will be set to the number of clients.

To run, build the C driver with -DENABLE_SSL=OPENSSL and set the target as the benchmark's name.

I used macOS's Instruments to do a time profile on the resulting executable. See the video: Profile libmongoc with Instruments.

Performance improvement

	Total runtime	Time in mongoc_openssl_ctx_new()	% Spent in mongoc_openssl_ctx_new()
Baseline	28.2s	11.81s	41.8%
With ssl_ctx sharing	2.9s	1.0 ms	0.034%