add compression

[mcginty]

Snappy or LZ4 comes to mind: https://github.com/BurntSushi/rust-snappy, https://github.com/bozaro/lz4-rs

[lunemec]

Ideal would be something that can compress the stream in multiple threads to achieve highest throughput. This could be tested for both libraries though.

[rmmh]

Note that the best compressor to use depends on the particular combination of compression speed, bandwidth, and decompression speed. Single-threaded zlib is better than LZ4 at <~5MB/s, ZSTD is the best at <32MB/s (but doesn't have a pure-rust implementation), etc.

Some more details and theory here: http://fastcompression.blogspot.com/2015/01/zstd-stronger-compression-algorithm.html

[mcginty]

The crypto is currently not much of a bottleneck, so I think it's also wondering how much we want to burn the laps of our users :).