FATAL ERROR: zstd uncompress failed with error code 10

[itsManjeet]

Describe the bug I am trying to compress a directory containing files of size ~= 2.5GiB via mksquashfs /input/path/* /path/to/output.img -comp zstd -b 256K -noappend -Xcompression-level 22

/path/to/output.img is inside a mounted s3fs directory (that's the possible cause of issue, but current architecture of the tool is reason to do that). however the same command work for data less than ~= 1 GiB.

To Reproduce Steps to reproduce the behavior:

1. Directory with more that 2.5 GiB data
2. A mounted s3fs bucket
3. The above mentioned command
4. FATAL ERROR: zstd uncompress failed with error code 10

Expected behavior This thing should work the same way it works with data lower than 1 GiB

Error Code FATAL ERROR: zstd uncompress failed with error code 10

Desktop (please complete the following information):

• OS: Ubuntu
• Version 22.04

Additional context Error is on myside for sure, I need help in debugging the error code as i am not much aware about the code base of ZSTD,

• I tried to get information of the exit code from manual but missing.

I also tried to use the ZSTD_getErrorName(10) for some meaningful information but it gave No error detected.

[figueroandy]

I am posting "me too" to this issue, as reported to bugs.gentoo.org on 4/16 and 4/17/2022 at: https://bugs.gentoo.org/838862

From 4/16/2022 Upon restoring or even reading content of very large tar.zstd archives (11 GB or larger) containing very large files (6 GB or larger) the following error results:

$ tar xf /mnt/backup3/data/vboxmnt.tar.zst ... "/stdin\ : Decoding error (36) : Restored data doesn't match checksum tar: Child returned status 1 tar: Error is not recoverable: exiting now"

The files appear to be extracted without error in spite of the error message.

Tested by masking app-arch/zstd-1.5.2 and re-emerging app-arch/zstd-1.5.0, re-creating the archive and then again extracting the archive as above, but without error message.

Archive was created with the following command: tar cpf /mnt/backup3/data/vboxmnt.tar.zst -I "zstd -9 -T0" mnt/vbox/VDI/

which contains several VirtualBox virtual machines.

Smaller files are not affected, and the error message appears to be passed to tar from zstd, and only shows up once after all files have been extracted.

and on 4/17/2022: Followup on issue with apparent zstd errors in my largest archives. Last night's archives were made with zstd-1.5.0 at my normal compression level 9, I continued to see an error upon testing the archive, but only on the largest archive in the collection of archives (total size of set of archives 93 GB at compression level 9), that one file being right at 30 GB. Simple test was done on the archive with $ zstd -t vboxmnt.tar.zst vboxmnt.tar.zst : 54.4 GiB... vboxmnt.tar.zst : Decoding error (36) : Corrupted block detected

and: $ tar tf vboxmnt.tar.zst mnt/vbox/VDI/ mnt/vbox/VDI/MXLinux.vdi mnt/vbox/VDI/MXLinuxFrugal.vdi mnt/vbox/VDI/Windows10Home.vdi /stdin\ : Decoding error (36) : Corrupted block detected tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now

Previously with version zstd-1.5.2, the error manifested as a checksum error.

I therefore reinstalled zstd-1.5.2 and recreated the collection of archives at the default compression level of 3. Retesting the collection of archives as above, turned up no errors.

Loss of compression between level 3 and level 9 was only about 5%, with a nearly 20% improvement in speed. That's a good trade off since I've moved to larger drives.

My conclusion is that the apparent errors is more the product of higher, but still modest, compression levels, more than the changes between versions, though the errors manifest more readily with version 1.5.2 than 1.5.0.

Strongly suspect the problem is upstream, though I've not been able to find reports of such errors by searching the web.

Note: It was suggested at that time that I post upstream, but had gotten distracted.

OS: Gentoo amd64 with kernel 5.10.148

[yoniko]

Hi and sorry for the delayed response. There's a good chance that there are two separate environment-related problems here.

@itsManjeet - error code 10 is unknown prefix which can happen when we are trying to read a Zstd frame and get bytes that do not match the magic -- either due to it being different or the read is shorter than 4 bytes.

1. Is it possible that reads return unexpectedly short buffers when using s3fs?
2. Does the problem still reproduce on the dev version of Zstd?

@figueroandy - I've tried reproducing locally and failed. A couple of followup questions:

1. Does it reproduce with binaries compiled from dev Zstd?
2. Can you try reading the files with zstd -t without using tar?
3. You mentioned The files appear to be extracted without error in spite of the error message.. Did you check that they are complete? if so this might indicate some sort of corrupted trailing frame being added.

[yoniko]

Issue https://github.com/facebook/zstd/issues/3350 may be related. EDIT: Probably not related as 3350 is only relevant for 32 bit binaries.

[terrelln]

error code 10 is unknown prefix which can happen when we are trying to read a Zstd frame and get bytes that do not match the magic -- either due to it being different or the read is shorter than 4 bytes.

It can also be thrown if there are unexpected trailing bytes after the zstd frame.

[yoniko]

Closing due to lack of information and reproducer. Please re-open when more relevant information surfaces.