fluent-plugin-s3 icon indicating copy to clipboard operation
fluent-plugin-s3 copied to clipboard

Add new ZST compression plugin

Open asvinours opened this issue 7 years ago • 4 comments

I added a new compression plugin based on facebook zstandard compression system.

zstandard offers faster and better compression then gzip

$ time zstd -f -5 random.log -o random.log.zst
Compressed 179309772 bytes into 21463467 bytes ==> 11.97%                      

real    0m1.584s
user    0m0.408s
sys     0m1.072s

$ time lzop -qf1 -o random.log.lzo random.log

real    0m0.387s
user    0m0.304s
sys     0m0.068s

$ time gzip -c random.log > random.log.gz

real    0m3.224s
user    0m2.916s
sys     0m0.180s

$ ls -lh
172M Sep  1 13:43 random.log
49M Sep  1 13:43 random.log.lzo
31M Sep  1 18:58 random.log.gz
21M Sep  1 18:57 random.log.zst

asvinours avatar Sep 01 '17 23:09 asvinours

Do other users still want this compression? Zst is now widely used so bundle it is reasonable.

repeatedly avatar Nov 15 '18 03:11 repeatedly

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days

github-actions[bot] avatar Jul 06 '21 10:07 github-actions[bot]

@asvinours

Oh, I've unexpectedly reflected merge commit. Could you reset https://github.com/fluent/fluent-plugin-s3/pull/205/commits/06c8bd21317a2ff1f2e0008c456f75eaf2c3a7a9 and rebase with the master, please? And follow( DCO git commit --sign )

kenhys avatar Jul 19 '21 05:07 kenhys

Any plans of merging this one?

siddharthg avatar Nov 21 '22 12:11 siddharthg