Avoid unnecessary fragmentation for small streams

[0x0ece]

Hi, I wanted to float an idea before possibly submitting a PR.

When sending a large number of small streams (e.g., smaller than MTU), Quinn currently tends to split streams into 2 frames across 2 packets. My proposal would be to explicitly avoid this behavior, sending the entire stream in a single frame, in a separated packet.

The change itself would be relatively small, kind of like this example placed after this line:

// Avoid fragmentation for small streams
let unsent_sz = (stream.pending.offset - stream.pending.unsent) as usize;
if unsent_sz <= 1200 && unsent_sz > max_buf_size {
    continue;
}

Here are some caveats:

• I think that the size should be part of the config, a good default may be 1200 but may depend on the use case
• stream.pending.offset and unsent are not visible in that fn, so we should probably add a method to stream.pending to retrieve the unsent_sz
• An additional condition could be stream.pending.unsent == 0, i.e. the entire stream is small, not just the remaining portion
• The behavior could be continue or break. I think continue makes more sense but I can also see valid reasons to break (Anyway this is just to say that I think that the change is small, but there’s a few “little” details that you’re probably best at deciding than me.)

---

As for the concrete use case / how we found the improvement. The Solana blockchain uses Quinn for ingesting new transactions. Transactions are relatively small (<= 1232 bytes, this would be the config I’d propose to use for Solana) and they’re sent creating a new stream per transaction (in principle, they could use datagrams, but using streams will make it easier in future to grow the transaction size).

We gathered some stats on 1h of traffic, during which we saw an average of 5k transactions/sec and of those ~15% were fragmented over 2 packets (note that the max tx size is 1232 but average is prob like 6-800 bytes). With the proposed change, transactions that don’t fit would spill in the next packet, but note that the total number of packets would be more or less the same. And, on the server side, there’d be no work to defrag.

In terms of resources, I don’t have numbers for Quinn, but we’re working on a custom server in C that only handles the specific case to defrag 1 tx split in 2 packets and that consumes ~1GB ram to sustain 5k tx/s — not trivial. Since Quinn server handles the generic case, I assume it’d require the same amount of resources or more.

In summary, I think that fragmenting small streams can be detrimental for use cases with many small streams, and avoiding that would be a relatively small change.

I’d like to hear your opinion and, as I mentioned, I’m happy to submit a PR if you think that’d help.