fst
fst copied to clipboard
Optionally reduce block and buffer sizes
As noted at https://github.com/richfitz/storr/pull/111#issuecomment-510237712, this could conserve memory without sacrificing speed, and it could improve load balancing among threads.
Thanks for submitting!
Yes, in general, we only need to allocate as many blocks as we have threads. Because currently the maximum block size is fairly large (1.7 GB), a large amount of RAM can be allocated during compression.
Such large blocks don't really help compression so it's better to reduce the block size as much as possible without sacrificing compression ratio (probably to a few tens of MB's).