Coalescing TLS records

[nametoolong]

This is an ad-hoc patch that tries to coalesce multiple TLS handshake records into one packet, so that handshakes are made slightly faster, especially for DTLS.

I am not familiar enough with Botan's code, not could I figure out how should I coalesce multiple TLS messages into a single record when their epochs interleave, so this patch just concatenates individual records. Its approach might be troublesome (exceptions are completely not handled), and it might have introduced memory hazards.

I would like to ask for such a feature to be added into the library instead of sending a badly-written PR, but I think submitting my patch might be helpful for other users. Please review with care.

EDIT: Alternatively we can implement this on a per-message basis for DTLS, as there is already a queue of messages in Datagram_Handshake_IO. Stream-oriented TLS can be left as-is.

[codecov-io]

Codecov Report

Merging #2284 into master will decrease coverage by <.01%. The diff coverage is 98.24%.

@@            Coverage Diff            @@
##           master   #2284      +/-   ##
=========================================
- Coverage    92.6%   92.6%   -0.01%     
=========================================
  Files         556     556              
  Lines       60626   60678      +52     
  Branches     6428    6435       +7     
=========================================
+ Hits        56145   56189      +44     
- Misses       4481    4489       +8

Impacted Files	Coverage Δ
src/lib/tls/tls_server.cpp	89% <100%> (+0.05%)	:arrow_up:
src/lib/tls/tls_client.cpp	93.11% <100%> (+0.04%)	:arrow_up:
src/lib/tls/tls_channel.cpp	91% <98.11%> (+0.97%)	:arrow_up:
src/lib/pk_pad/emsa_raw/emsa_raw.cpp	71.42% <0%> (-2.86%)	:arrow_down:
src/lib/pubkey/dl_group/dl_group.cpp	92.12% <0%> (-1.19%)	:arrow_down:
src/lib/pubkey/mce/mceliece_key.cpp	85.39% <0%> (-1.13%)	:arrow_down:
src/bogo_shim/bogo_shim.cpp	88.77% <0%> (-0.36%)	:arrow_down:
src/lib/misc/cryptobox/cryptobox.cpp	95.23% <0%> (+1.58%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cb6f4c4...cb21fbe. Read the comment docs.

[randombit]

Thanks. Right now we rely heavily on transport layer (ie whatever the IO callback layer is doing) to coalesce writes. That works ok for TCP most of the time but is certainly not optimal for DTLS.

I'm sick and not really in any condition to review a patch touching code this complicated right now but I'll take a look soon and hopefully it can either be made to work or at least inspire another implementation.

[codecov-commenter]

Codecov Report

Merging #2284 (cb21fbe) into master (432a659) will decrease coverage by 0.00%. The diff coverage is 98.24%.

:exclamation: Current head cb21fbe differs from pull request most recent head fe65d24. Consider uploading reports for the commit fe65d24 to get more accurate results

@@            Coverage Diff             @@
##           master    #2284      +/-   ##
==========================================
- Coverage   92.60%   92.60%   -0.01%     
==========================================
  Files         556      556              
  Lines       60626    60678      +52     
  Branches     6428     6435       +7     
==========================================
+ Hits        56145    56189      +44     
- Misses       4481     4489       +8     

Impacted Files	Coverage Δ
src/lib/tls/tls_channel.cpp	91.00% <98.11%> (+0.97%)	:arrow_up:
src/lib/tls/tls_client.cpp	93.11% <100.00%> (+0.04%)	:arrow_up:
src/lib/tls/tls_server.cpp	89.00% <100.00%> (+0.05%)	:arrow_up:
src/lib/pk_pad/emsa_raw/emsa_raw.cpp	71.42% <0.00%> (-2.86%)	:arrow_down:
src/lib/pubkey/dl_group/dl_group.cpp	92.12% <0.00%> (-1.19%)	:arrow_down:
src/lib/pubkey/mce/mceliece_key.cpp	85.39% <0.00%> (-1.13%)	:arrow_down:
src/bogo_shim/bogo_shim.cpp	88.77% <0.00%> (-0.36%)	:arrow_down:
src/lib/misc/cryptobox/cryptobox.cpp	95.23% <0.00%> (+1.58%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cb6f4c4...fe65d24. Read the comment docs.

[nametoolong]

I should also note some 802.1x protocols like PEAP or EAP-TTLS use stream TLS but in a request/response way. Write coalescence is a must for EAP supplicants.

[reneme]

FWIW: The TLS 1.3 implementation (currently under active development) does implement coalescing of handshake messages into single records: https://github.com/randombit/botan/blob/ffe1c30395ed2c6f8d7c89e024185f1c7513b46f/src/lib/tls/tls13/tls_channel_impl_13.h#L26-L66