Reduce memory usage of s2n_connection

[fatrat1117]

Problem:

Currently, the size of s2n_connection is about 14.5K thus about 14G for 1 million connections on a high throughput server.

(gdb) p sizeof(struct s2n_connection) $1 = 14656

The server could suffer from OOM issue potentially because of it. We need to find a way to move memory that is not frequently accessed or is only needed for the negotiation off of the s2n_connection allocation.

Proposed Solution:

For example, not every connection needs ALPN, we could create following variable when necessary. char application_protocol[256];

The idea is to identify large and less frequent used variable/struct and create them as needed.

[colmmacc]

I'd love if we could split the memory we need during negotiation from the memory we need during a connection ... but especially in a way that could avoid dynamic allocation. Callers could keep a pool of objects for handshakes, and a pool for connections. That way they can be right-sized for each service, based on new-connections-per-second and average duration of each connection. Allocating and unallocating a lot would degrade performance.

[lrstewart]

sizeof(struct s2n_connection) is currently ~4k. We're still (and probably always) looking to improve, but I'm going to close this ticket as out of date.