spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

[FEA][Spark 3.5.0] Add support for lz4raw compression in Parquet files

Open andygrove opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe. Spark 3.5.0 adds support for lz4raw compression in Parquet files. cuDF does not support this, so we cannot support this on GPU.

Describe the solution you'd like Add support for this compression codec.

Describe alternatives you've considered

Additional context

andygrove avatar Aug 31 '23 15:08 andygrove

Need to document limitation for lz4raw compression. File issues for downstream teams for support.

Ref: https://issues.apache.org/jira/browse/PARQUET-2032.

mattahrens avatar Sep 06 '23 20:09 mattahrens

Added an issue in cuda-hpc-libraries project in gitlab-master

razajafri avatar Apr 01 '24 15:04 razajafri