zstd compression output differens with the same options between 1.5.5 and 1.5.6

[djugei]

Description and reproduction equal: sha1sum libnghttp2-1.62.0-1-x86_64.pkg.tar.zst zstdcat libnghttp2-1.62.0-1-x86_64.pkg.tar.zst | zstd-1.5.6 -c -T0 --ultra -20 - | sha1sum

different: sha1sum libnghttp2-1.62.0-1-x86_64.pkg.tar.zst zstdcat libnghttp2-1.62.0-1-x86_64.pkg.tar.zst | zstd-1.5.5 -c -T0 --ultra -20 - | sha1sum

libnghttp2-1.62.0-1-x86_64.pkg.tar.zst is an archlinux package that has been compressed with 1.5.6

Expected behavior either stable output of the cli tool, or documentation that the output is not stable

[djugei]

i assume that this is related to this part of the changelog: perf: improved compression of arrays of integers at high compression, by @Cyan4973

[christian-heusel]

this is a bit of a reproduction issue as now the signatures of the arch packages only match if you use the correct (unspecified) zstd version

This version is recorded in the .BUILDINFO as part of the package, so its only an issue for your setup and not for the official Arch Linux reproducible stuff 😊

[djugei]

This version is recorded in the .BUILDINFO as part of the package, so its only an issue for your setup and not for the official Arch Linux reproducible stuff 😊

i opened this issue before finding that out, still means you need arch testing to reproduce arch stable. but thats an arch issue not a zstd issue. i removed that part from the initial issue description

[Cyan4973]

There is no guarantee of output reproducibility between versions. If you want reproducible compressed artifacts, you need to use the exact same version. There is nothing specific. No one expects gcc-13 to generate the same binary as gcc-9 for example. Keeping versions identical between builds is standard requirement of any reproducible toolchain.

[djugei]

yeah that makes a lot of sense, just caught me a bit off guard :)