gcs-cacher icon indicating copy to clipboard operation
gcs-cacher copied to clipboard

Add ability to skip compression when making a tar

Open mandrizzle opened this issue 3 years ago • 1 comments

I have been doing some performance testing with this library and found depending on the directory, eg caching binary files, compression does little to reduce the size. It however can take quite a long time for it to finish.

In my testing, a folder with 600MB of binary files tar -czf 600MBFolder/ ~1 minute | Output size ~600MB

tar -cf 600MBFolder/ ~5 seconds | Output size ~600MB

That being said, I was thinking adding a flag to optionally skip compression on a directory may be quite useful

mandrizzle avatar Mar 04 '21 20:03 mandrizzle

It would also be helpful if we could provide a custom compression command, I found zstd to be a lot faster than gzip. It can be used like this:

compress

tar -I "zstd -T2 --fast=3" -cf ...

decompress

tar -I "zstd -d -T2" -xf ...

JCMais avatar Jan 14 '22 12:01 JCMais