fst icon indicating copy to clipboard operation
fst copied to clipboard

Optionally reduce block and buffer sizes

Open wlandau opened this issue 6 years ago • 1 comments

As noted at https://github.com/richfitz/storr/pull/111#issuecomment-510237712, this could conserve memory without sacrificing speed, and it could improve load balancing among threads.

wlandau avatar Jul 12 '19 16:07 wlandau

Thanks for submitting!

Yes, in general, we only need to allocate as many blocks as we have threads. Because currently the maximum block size is fairly large (1.7 GB), a large amount of RAM can be allocated during compression.

Such large blocks don't really help compression so it's better to reduce the block size as much as possible without sacrificing compression ratio (probably to a few tens of MB's).

MarcusKlik avatar Jul 12 '19 21:07 MarcusKlik