zstd-rs icon indicating copy to clipboard operation
zstd-rs copied to clipboard

zstd::bulk::decompress(.., capacity) immediately allocates a Vec that size

Open dw opened this issue 9 months ago • 0 comments

Hey there,

I misinterpreted this parameter to mean a maximum amount of data to decompress, instead it is directly controlling a Vec allocation, resulting in a surprising OOM much later, and possibly also a hidden perf issue as system calls and (kernel mmap_sem etc locking) will likely be involved for any large value.

The unused Vec capacity is also not trimmed. It's possible (though I guess unlikely) with a user running this in a loop for many small strings but a large capacity to discover either slowness or surprising breakage.

Would suggest perhaps documenting the dual meaning of capacity. I'm not sure what a more desirable behaviour would be - arguably the current behaviour is fine, particularly for cases where the only alternative would be incremental reallocation and memmoves killing some of the perf benefit to using zstd at all.

Thanks for a great library

dw avatar Feb 24 '25 11:02 dw