dspeterson
zstd (Zstandard) compression
After reading this: https://blog.cloudflare.com/squeezing-the-firehose/ I wonder if zstd is or becomes an option in dory?
It definitely looks like something that would be good to add to dory. I added a to-do item for zstd to dory's wiki page http://dory.wikidot.com/start . I also took a look at KIP-110 which covers zstd compression for Kafka: https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression . Based on the information there, it looks like adding zstd support to dory would require two areas of work:
- Updating Dory to support Kafka's v2 wire protocol. This is needed because according to KIP-110, Kafka only supports zstd compression for v2 clients.
- Adding a compression codec for zstd to dory. Compression codecs live here: https://github.com/dspeterson/dory/tree/master/src/dory/compress
Most of the work is in item 1. I'd call that a medium-sized project. I've been intending to add v2 wire protocol support to dory for a while and hope to do it in the not too distant future although it's hard to predict when. Once item 1 is done, item 2 should be very easy.
Looking forward to this although I can see it could be a lot of work!
On another note, does v2 wire protocol give any other advantages that dory is missing atm?
The v2 protocol adds some changes that support exactly once delivery semantics, as well as a timestamp field. There may be some other things I'm leaving out. I haven't yet had a chance to look at it in depth.