facebook
Allocate memory dynamically
Is your feature request related to a problem? Please describe.
zst reserves the entire specified memory shortly after starting, even if the amount of processed data is nowhere close.
Describe the solution you'd like If possible, I suggest only allocating as much memory as needed at a time, rather than immediately reserving the entire specified memory.
Describe alternatives you've considered This is also the behaviour of LZMA-based compressors (lzma, xz, 7z).
Note that, even though zstd can allocate a buffer with a size that is proportional to the window size, the buffer is only allocated, and not initialized.
In most modern Operating System, it means no real memory is really reserved for the buffer at the moment it's allocated, right until it is actually needed.
This should cover many scenarios where the library is blind and must allocate in advance the maximum possible memory in case it needs it (streaming mostly).
But also note that, in many other scenarios, where exact needs are known upfront, zstd is smart enough to only reserve what it needs. For example, if one requires a 1 GB window size, and then provides a 1 KB file to compress, zstd will notice the discrepancy, and automatically size down the requirements.
