bin: use highest zstd compression level

Open Chocimier opened this issue 5 years ago • 4 comments

Files are few percent smaller, decompression is still instant.

Apr 30 '20 20:04 Chocimier

I think this can use a lot of memory during compression?

Also, compression takes significantly longer, see https://community.centminmod.com/data/attachment-files/2017/09/5843_compress-test-zstd1.3.1-threaded-1-19-030917.png

Apr 30 '20 20:04 leahneukirchen

Our largest package files are, I believe, 2 gigabytes (some noarch data pack for some possibly restricted game, maldridge as maintainer). What's the memory consumption when compressing those?

May 01 '20 00:05 Vaelatern

zstd 9: 24-180 M zstd 19: 100-270 M xz 9 (4 cores): 520-820 M

Measured on flightgear-data, linux5.6-dbg and kicad-packages3D (last is biggest unpacked but highly compressible). details

May 01 '20 17:05 Chocimier

Level 21 can run out of memory on 32bit machines: https://github.com/facebook/zstd/pull/2711

Idk how close to it 19 can be.

Jun 27 '21 21:06 ericonr