python-zstandard icon indicating copy to clipboard operation
python-zstandard copied to clipboard

Feature request: allow `decompress` to decompress stream data without content size in the header

Open CaselIT opened this issue 4 years ago • 1 comments

Edit: this is documented, but I think it would be an useful addition. Maybe by passing max_output_size=-1?

The example below raises ZstdError: could not determine content size in frame header

import io
import zstandard as zstd
data = b'some data'
input_buffer = io.BytesIO(data)
with zstd.ZstdCompressor().stream_reader(input_buffer) as r:
    compressed = r.read()
decompressed = zstd.decompress(compressed)
assert decompressed == data

Passing max_output_size to the decompress call works as expected:

...
decompressed = zstd.decompress(compressed, max_output_size=len(data))
assert decompressed == data

Using ZstdDecompressor.stream_reader also works:

...
with zstd.ZstdDecompressor().stream_reader(io.BytesIO(compressed)) as r:
    decompressed = r.read()
assert decompressed == data

I think decompress should handle this case as well, since it's not always possible to known how the data was compressed.

The package version I use is zstandard 0.15.2 on windows 10 (but I don't think it's os dependent)

CaselIT avatar Apr 26 '21 10:04 CaselIT

Ok reading a the documentation in more details, this is the expected behaviour: https://github.com/indygreg/python-zstandard/blob/477776e6019478ca1c0b5777b073afbec70975f5/zstandard/backend_cffi.py#L3656-L3661

This is then a feature request, since I think it would be an useful addition.

CaselIT avatar Apr 26 '21 10:04 CaselIT