gcs-cacher
gcs-cacher copied to clipboard
Add ability to skip compression when making a tar
I have been doing some performance testing with this library and found depending on the directory, eg caching binary files, compression does little to reduce the size. It however can take quite a long time for it to finish.
In my testing, a folder with 600MB of binary files
tar -czf 600MBFolder/
~1 minute | Output size ~600MB
tar -cf 600MBFolder/
~5 seconds | Output size ~600MB
That being said, I was thinking adding a flag to optionally skip compression on a directory may be quite useful
It would also be helpful if we could provide a custom compression command, I found zstd to be a lot faster than gzip. It can be used like this:
compress
tar -I "zstd -T2 --fast=3" -cf ...
decompress
tar -I "zstd -d -T2" -xf ...