polars icon indicating copy to clipboard operation
polars copied to clipboard

perf(python,rust): Use faster compression backend for parquet

Open ritchie46 opened this issue 3 years ago • 3 comments

This uses zlib-ng and lz4_flex as backend in favor of the defaults. #4259

ritchie46 avatar Aug 04 '22 18:08 ritchie46

Tests fail, so have to investigate later.

ritchie46 avatar Aug 04 '22 19:08 ritchie46

Codecov Report

Merging #4262 (528a0da) into master (d1e5b10) will increase coverage by 0.64%. The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #4262      +/-   ##
==========================================
+ Coverage   64.05%   64.70%   +0.64%     
==========================================
  Files         462      460       -2     
  Lines       76112    75314     -798     
==========================================
- Hits        48757    48734      -23     
+ Misses      27355    26580     -775     
Impacted Files Coverage Δ
polars/polars-core/src/vector_hasher.rs 74.19% <0.00%> (ø)
polars/polars-lazy/src/frame/parquet.rs 53.48% <0.00%> (-11.63%) :arrow_down:
...s/polars-core/src/series/implementations/floats.rs 59.50% <0.00%> (-0.93%) :arrow_down:
py-polars/src/dataframe.rs 88.67% <0.00%> (-0.80%) :arrow_down:
polars/polars-core/src/fmt.rs 45.60% <0.00%> (-0.74%) :arrow_down:
...rs-lazy/src/logical_plan/optimizer/file_caching.rs 95.28% <0.00%> (-0.59%) :arrow_down:
...olars/polars-core/src/frame/groupby/into_groups.rs 59.58% <0.00%> (-0.30%) :arrow_down:
...lars/polars-core/src/chunked_array/builder/list.rs 62.29% <0.00%> (-0.28%) :arrow_down:
polars/polars-io/src/csv/buffer.rs 79.55% <0.00%> (-0.25%) :arrow_down:
py-polars/polars/internals/frame.py 91.37% <0.00%> (-0.21%) :arrow_down:
... and 12 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov-commenter avatar Aug 04 '22 19:08 codecov-commenter

Keep in mind that there are alsos some kind of regression in zlib-ng related to gzipped CSV files (or at least with the rust binding). https://github.com/rust-lang/libz-sys/issues/104

ghuls avatar Sep 09 '22 10:09 ghuls