zstd
zstd copied to clipboard
Extra verbose output prints wrong value for `wlog` when doing `zstd -19 --long`
How to reproduce:
Here is me trying to compress the same pipeline. First with -19
:
$ cat log.txt | zstd -c -vv -19 > log.txt.19.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
--zstd=wlog=23,clog=24,hlog=22,slog=7,mml=3,tlen=256,strat=9
--format=.zst --block-size=0 --memory=134217728 --threads=1 --content-size
Using stdin for input
Using stdout for output
Sparse File Support is automatically disabled on stdout ; try --sparse
Decompression will require 8388608 B of memory
/*stdin*\ : Completed in 7.92 sec (cpu load : 99%) 2.92%
$ zstd -vv -l log.txt.19.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
log.txt.19.zst
# Zstandard Frames: 1
Window Size: 8388608 B (8388608 B)
Compressed Size: 619522 B (619522 B)
Check: XXH64
Note that the verbose output reports wlog=23
(8MB) which agrees to the output of zstd -vv -l
.
If I run the compression command with -19 --long
I get the exact same verbose output (wlog=23
) but the compression result is different and zstd -vv -l
reports 128MB.
$ cat log.txt | zstd -c -vv -19 --long > log.txt.19long.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
--zstd=wlog=23,clog=24,hlog=22,slog=7,mml=3,tlen=256,strat=9
--format=.zst --block-size=0 --memory=134217728 --threads=1 --content-size
Using stdin for input
Using stdout for output
Sparse File Support is automatically disabled on stdout ; try --sparse
Decompression will require 8388608 B of memory
/*stdin*\ : Completed in 9.18 sec (cpu load : 99%) 2.91%
$ zstd -vv -l log.txt.19long.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
log.txt.19long.zst
# Zstandard Frames: 1
Window Size: 134217728 B (134217728 B)
Compressed Size: 616714 B (616714 B)
Check: XXH64
Version info and OS
$ zstd --version
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
$ uname -a
Linux OpenSUSE 5.17.7-1-default #1 SMP PREEMPT Thu May 12 12:38:04 UTC 2022 (c9a5fa1) x86_64 x86_64 x86_64 GNU/Linux
One thing we can do here would be to add a test that runs a compression, captures the flag outputs, reruns the compression under those flags, and then validates that the output is identical. It could be a partial validation going forward that this logic is staying in sync.