zstd icon indicating copy to clipboard operation
zstd copied to clipboard

Extra verbose output prints wrong value for `wlog` when doing `zstd -19 --long`

Open jimis opened this issue 2 years ago • 1 comments

How to reproduce:

Here is me trying to compress the same pipeline. First with -19:

$ cat log.txt  | zstd -c -vv -19 > log.txt.19.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
--zstd=wlog=23,clog=24,hlog=22,slog=7,mml=3,tlen=256,strat=9
--format=.zst --block-size=0 --memory=134217728 --threads=1 --content-size
Using stdin for input 
Using stdout for output 
Sparse File Support is automatically disabled on stdout ; try --sparse 
Decompression will require 8388608 B of memory
/*stdin*\            : Completed in 7.92 sec  (cpu load : 99%)                  2.92%  
$ zstd -vv -l log.txt.19.zst 
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
log.txt.19.zst 
# Zstandard Frames: 1
Window Size: 8388608 B (8388608 B)
Compressed Size: 619522 B (619522 B)
Check: XXH64

Note that the verbose output reports wlog=23 (8MB) which agrees to the output of zstd -vv -l.

If I run the compression command with -19 --long I get the exact same verbose output (wlog=23) but the compression result is different and zstd -vv -l reports 128MB.

$ cat log.txt  | zstd -c -vv -19 --long > log.txt.19long.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
--zstd=wlog=23,clog=24,hlog=22,slog=7,mml=3,tlen=256,strat=9
--format=.zst --block-size=0 --memory=134217728 --threads=1 --content-size
Using stdin for input 
Using stdout for output 
Sparse File Support is automatically disabled on stdout ; try --sparse 
Decompression will require 8388608 B of memory
/*stdin*\            : Completed in 9.18 sec  (cpu load : 99%)                  2.91%  
$ zstd -vv -l log.txt.19long.zst 
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
log.txt.19long.zst 
# Zstandard Frames: 1
Window Size: 134217728 B (134217728 B)
Compressed Size: 616714 B (616714 B)
Check: XXH64

Version info and OS

$ zstd --version
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
$ uname -a
Linux OpenSUSE 5.17.7-1-default #1 SMP PREEMPT Thu May 12 12:38:04 UTC 2022 (c9a5fa1) x86_64 x86_64 x86_64 GNU/Linux

jimis avatar May 24 '22 13:05 jimis

One thing we can do here would be to add a test that runs a compression, captures the flag outputs, reruns the compression under those flags, and then validates that the output is identical. It could be a partial validation going forward that this logic is staying in sync.

felixhandte avatar Jun 02 '22 17:06 felixhandte