cc-index-table icon indicating copy to clipboard operation
cc-index-table copied to clipboard

Published 20 hours ago • verified publisher icon commoncrawl

Explore Zstandard compression

Open sebastian-nagel opened this issue 2 years ago • 1 comments

(depends on #10)

Zstandard compression is directly integrated into Parquet (PARQUET-1866). Also Athena now supports Zstd as Parquet compression. Time to explore whether switching from gzip to zstd brings improvements in storage size or query processing speed. In addition to Athena and Spark, support for zstd should be verified also Hive, Arrow, Pandas and other common tools to read/process Parquet files.

sebastian-nagel avatar Dec 08 '21 19:12 sebastian-nagel

By passing --outputCompression=zstd the Parquet files are compressed using Zstandard compression. However, the zstd-compressed files are slightly bigger compared to gzip. There is no effect on the Parquet file size visible by adjusting the zstd compression level (eg. --conf spark.io.compression.zstd.level=22) - this option might only affect other output formats.

sebastian-nagel avatar May 31 '22 10:05 sebastian-nagel