fluent-plugin-s3
fluent-plugin-s3 copied to clipboard
Add new ZST compression plugin
I added a new compression plugin based on facebook zstandard compression system.
zstandard offers faster and better compression then gzip
$ time zstd -f -5 random.log -o random.log.zst
Compressed 179309772 bytes into 21463467 bytes ==> 11.97%
real 0m1.584s
user 0m0.408s
sys 0m1.072s
$ time lzop -qf1 -o random.log.lzo random.log
real 0m0.387s
user 0m0.304s
sys 0m0.068s
$ time gzip -c random.log > random.log.gz
real 0m3.224s
user 0m2.916s
sys 0m0.180s
$ ls -lh
172M Sep 1 13:43 random.log
49M Sep 1 13:43 random.log.lzo
31M Sep 1 18:58 random.log.gz
21M Sep 1 18:57 random.log.zst
Do other users still want this compression? Zst is now widely used so bundle it is reasonable.
This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days
@asvinours
Oh, I've unexpectedly reflected merge commit. Could you reset https://github.com/fluent/fluent-plugin-s3/pull/205/commits/06c8bd21317a2ff1f2e0008c456f75eaf2c3a7a9 and rebase with the master, please? And follow( DCO git commit --sign )
Any plans of merging this one?