spark-rapids
spark-rapids copied to clipboard
[FEA][Spark 3.5.0] Add support for lz4raw compression in Parquet files
Is your feature request related to a problem? Please describe. Spark 3.5.0 adds support for lz4raw compression in Parquet files. cuDF does not support this, so we cannot support this on GPU.
Describe the solution you'd like Add support for this compression codec.
Describe alternatives you've considered
Additional context
Need to document limitation for lz4raw compression. File issues for downstream teams for support.
Ref: https://issues.apache.org/jira/browse/PARQUET-2032.
Added an issue in cuda-hpc-libraries project in gitlab-master