polars
polars copied to clipboard
perf(python,rust): Use faster compression backend for parquet
This uses zlib-ng and lz4_flex as backend in favor of the defaults. #4259
Tests fail, so have to investigate later.
Codecov Report
Merging #4262 (528a0da) into master (d1e5b10) will increase coverage by
0.64%. The diff coverage is0.00%.
@@ Coverage Diff @@
## master #4262 +/- ##
==========================================
+ Coverage 64.05% 64.70% +0.64%
==========================================
Files 462 460 -2
Lines 76112 75314 -798
==========================================
- Hits 48757 48734 -23
+ Misses 27355 26580 -775
| Impacted Files | Coverage Δ | |
|---|---|---|
| polars/polars-core/src/vector_hasher.rs | 74.19% <0.00%> (ø) |
|
| polars/polars-lazy/src/frame/parquet.rs | 53.48% <0.00%> (-11.63%) |
:arrow_down: |
| ...s/polars-core/src/series/implementations/floats.rs | 59.50% <0.00%> (-0.93%) |
:arrow_down: |
| py-polars/src/dataframe.rs | 88.67% <0.00%> (-0.80%) |
:arrow_down: |
| polars/polars-core/src/fmt.rs | 45.60% <0.00%> (-0.74%) |
:arrow_down: |
| ...rs-lazy/src/logical_plan/optimizer/file_caching.rs | 95.28% <0.00%> (-0.59%) |
:arrow_down: |
| ...olars/polars-core/src/frame/groupby/into_groups.rs | 59.58% <0.00%> (-0.30%) |
:arrow_down: |
| ...lars/polars-core/src/chunked_array/builder/list.rs | 62.29% <0.00%> (-0.28%) |
:arrow_down: |
| polars/polars-io/src/csv/buffer.rs | 79.55% <0.00%> (-0.25%) |
:arrow_down: |
| py-polars/polars/internals/frame.py | 91.37% <0.00%> (-0.21%) |
:arrow_down: |
| ... and 12 more |
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.
Keep in mind that there are alsos some kind of regression in zlib-ng related to gzipped CSV files (or at least with the rust binding). https://github.com/rust-lang/libz-sys/issues/104