miniz_oxide icon indicating copy to clipboard operation
miniz_oxide copied to clipboard

Compression level zero is very slow

Open fintelia opened this issue 1 year ago • 1 comments

When set to not apply compression, writing compressed output should mostly just amount to memcpy'ing the input data to the output. However, actual performance seem to be on the order of 120 MB/s. Some quick profiling indicates that a substantial amount of time is spent updating hash chains and recording literals. Neither of these operations should be necessary to produce uncompressed output.

Benchmarking via flate2:

let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(0));
encoder.write_all(&bytes).unwrap();
encoder.flush_finish().unwrap();

fintelia avatar Sep 21 '24 20:09 fintelia

I'm not done fixing it yet as it's still an order of magnitude slower than it should be, but I've started doing some work on it to split the stored compression mode to it's own function to skip the non-needed parts, and it's it seems to be something like twice as fast in the version I just published (0.8.3) compared to the previous versions.

It looks like the orig miniz didn't really care about making the stored mode performant and it's just a quick addon in the main compression function and thus does a lot of redundant stuff like you noted. I.e as ending up adding stuff to the lz dictionary even though that's pointless, and adding literals to the hash chain. Zlib just stores how many bytes that would be needed to be added which allows it to just defer it to the the non-stored compression function to add stuff from the backbuffer to the hash chain to be able to compress better if the compression mode is switched mid stream. (no idea in what situation one would change compression mode mid stream but it is a thing that is supported for whatever reason).

oyvindln avatar Jan 13 '25 20:01 oyvindln