Inefficient tls record creation with repeated small writes

[raycoll]

trafficstars

Consider the following example:

uint8_t data_buffer[1024];
while(true) {
    /* Prepare some data to write to s2n */
    const ssize_t fill_buffer_rc = fill_buffer(data_buffer);
    if (fill_buffer_rc <= 0) {
        break;
    }
    
    const int s2n_send_rc = s2n_send(s2n_conn, data_buffer, fill_buffer_rc);
    if (s2n_errno != S2N_ERR_T_BLOCKED) {
        break;
    }
}

Problem

In this example a limited application buffer size(1024 bytes) is used. Each call to s2n_send will result in a 1024 byte or less TLS record produced. In scenarios where the application has more than 1024 total bytes “immediately” available, this results in inefficient record creation in s2n.

Consider the case where repeated calls to fill_buffer would return up to 12000 bytes before erroring. The current s2n behavior is to create 12 TLS records from 12 calls to s2n_send.

For optimal throughput, the ideal behavior would be for s2n send to create one record with a 12000 byte fragment size(the maximum data the application has immediately available).

Suggested Solution

This throughput problem caused by TLS record creation is similar to the problem TCP_CORK solves at the TCP segment level. s2n could add a way to set a cork for TLS record creation: S2N_CORK ? Or any solution where the application can avoid small record writes without creating/populating a MAX_FRAGMENT_SIZE buffer.

oversimplified pseudocode(ignores all of the current buffering and sendv logic s2n has):

def s2n_conn_is_corked(conn):
     return conn.cork

def s2n_conn_set_cork(conn):
     conn.cork = true

def s2n_send(conn, data, len):
    if s2n_conn_is_corked():
        if len + conn.get_pending_data() > s2n_get_cork_ceiling():
            /* max data queued, time to flush */
            return s2n_create_record_and_flush(conn, data, len)
    else:
         return s2n_create_record_and_flush(conn, data, len)  

[Matthias247]

Instead of adding more wrappers around OS IO, consider to go the other way around just don't do IO at all. the application can grab encoded data and, send it whenever it thinks its the right time to do that and signal back to s2n when the data was sent.

Or s2n writes to an in-memory buffer which then has to be flushed by the application. Obviously those approaches might come with the cost of an extra data copy - but it can be avoided if you can make it possible to move the ownership of the buffer containing encoded data to the application.

That would also be a requirement for supporting any other IO means than traditional (nonblocking) sockets - e.g. completion based IO.

[raycoll]

+1 on a more flexible I/O paradigm. This issue is more about adding a mechanism to s2n that gives hints about when s2n should start creating tls records from input data so we don't end up with small encoded chunks(records) on repeated writes. I think we would want to support both the current I/O api and the future one.

[Matthias247]

It could also be left to the user. They can query how much data is encoded, and if they think its enough then start a flush() like operation, which encodes everything. In addition to that buffered data could be auto-encoded once a full-frame worth of data exists.

[raycoll]

You're right. Solution definitely need a flush()-like api so the application can tell s2n to encode/encrypt/etc the accumulated pending data.