Efficient decompression when the uncompressed size is unknown

[traceflight]

Hi,

I have some compressed data but I don’t know the minimum possible size after decompression. Are there any efficient ways to decompress it without knowing the exact uncompressed length?

Thanks.

[PSeitz]

You could over-allocate (maximum compression is 250x) an unitialized buffer and decompress into it. As long as the memory is never touched it should be cheap, since pages in the OS are usually lazy initialized. It maybe backfire though if the pages are not lazy initialized.

[traceflight]

thank you so much.

[traceflight]

I'd like to know if we can define a VecSink which is backed by a Vec<u8>. We can dynamically allocate memory when decompressing. I know this may cause poor performance, but it fits my use case. And we can conservatively guess a uncompressed size to avoid too many allocations.