datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Respect Spark compression settings in native Parquet writer

Open andygrove opened this issue 1 month ago • 0 comments

What is the problem the feature request solves?

https://github.com/apache/datafusion-comet/pull/2812 adds basic support for native Parquet writes, but does not fully support compression codec configs from Spark. Also, lz4 compression level is not configurable.

Describe the potential solution

Respect Spark configuration options.

Image

Also add a Comet conf for specifying lz4 compression level.

Additional context

No response

andygrove avatar Nov 21 '25 21:11 andygrove