kinesis-vcr
kinesis-vcr copied to clipboard
scopely
Compression
It does not; it currently tries to be extremely simple and just writes the bytes from the wire to S3 and reads them back.
Might make sense to compress, of course. I haven't found write performance to be a problem-- we keep up with 2 MB / sec / shard emission to S3 quite easily.
I'd be quite amenable to a PR that emits to .gz files and decompresses such files on the fly.
Definitely would prefer to compress - from the cost point of view, and reliability as well (less dependency on guaranteed write throughput). For us it's all about ensuring we have means to recover from errors. Will give it a try and if it suits us will certainly make the changes, can't guarantee it to be soon though. Thanks for useful project. Denis From: Avram Lyon [email protected] To: scopely/kinesis-vcr [email protected] Cc: Denis Mikhalkin [email protected] Sent: Tuesday, 6 October 2015, 10:48 Subject: Re: [kinesis-vcr] Compression (#32)
It does not; it currently tries to be extremely simple and just writes the bytes from the wire to S3 and reads them back.Might make sense to compress, of course. I haven't found write performance to be a problem-- we keep up with 2 MB / sec / shard emission to S3 quite easily.I'd be quite amenable to a PR that emits to .gz files and decompresses such files on the fly.— Reply to this email directly or view it on GitHub.
Having talked a bit about this with @casidiablo here at Scopely, we're interested. One of us might just do that ourselves.
Thanks for the idea!
We ended up doing the compression application-side and are no longer likely to do this at Scopely, since our streams themselves are usually compressed.