pixz icon indicating copy to clipboard operation
pixz copied to clipboard

Auto-optimize block size based on total file size

Open bremner opened this issue 11 years ago • 2 comments

On the file

http://d-i.debian.org/daily-images/amd64/20140214-00:18/netboot/mini.iso

pixz -p 8 mini.iso

only uses 2 threads.

It takes about 7 seconds on my i7, so it seems like there is room to ramp up.

bremner avatar Feb 14 '14 15:02 bremner

Thanks for the report!

The default compression level is -6, which uses a dictionary size of 8 MiB. The default block size ratio is 2.0, so that yields a block size of 16 MiB. Your mini.iso file is around 25 MiB, so it will only have two blocks that can be independently compressed, and therefore only two compression threads. You can verify this by running xz -lv mini.iso.xz, which should show just two blocks in the compressed file.

You can raise the number of threads available by lowering the compression level or the block size ratio, but of course this will make the compression worse. Eg: ``pixz -p 8 -2 -f 1.5 mini.iso` will use a block size of 3 MiB, and will exercise all eight threads.

It might be worthwhile for pixz to automagically optimize this when a small file is detected, which is in the TODO list: optimized settings -> block size, for max threads on small files. I'm not convinced, however, that pixz should be automatically causing worse compression than the user expects. Perhaps a small wrapper program would be useful for this, to call pixz with the parameters that you prefer.

vasi avatar Feb 16 '14 15:02 vasi

See also: https://github.com/vasi/pixz/issues/9

vasi avatar May 19 '14 16:05 vasi