incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Feature] Support zstd compression

Open zuston opened this issue 3 years ago • 1 comments

Motivation

ZSTD has a good tradeoff between compression ratio and de/compress speed. For reducing the shuffle-data stored size, it's necessary to support this compression algorithm.

How to do

  1. Introduce the abstract interface of de/compress method
  2. Support zstd

zuston avatar Aug 26 '22 10:08 zuston

Good idea.

jerqi avatar Aug 26 '22 11:08 jerqi

You can raise some other issues for other compression codec , such as snappy, as good first issues. Let more people can join our community more easily.

jerqi avatar Oct 26 '22 11:10 jerqi

close by #254

jerqi avatar Oct 26 '22 11:10 jerqi