TurboPFor-Integer-Compression
TurboPFor-Integer-Compression copied to clipboard
Please document slop requirements
Hi,
It seems that the PFor functions can read bytes past the end of the posting list, and write more output values than “num”. In particular, 32 values seem to be decoded at a time even if num == 1. This means the application must be careful to allow some slop to avoid crashes. Is the amount of required slop documented anywhere, ie., M extra bytes of input and space for N output values must always be available?
some of it is "documented" in the maintainers answers here: https://github.com/powturbo/TurboPFor-Integer-Compression/issues/26
I've made changes, so that there must be nor more reading/writing outside the i/o buffers without loosing any speed. More tests are necessarry for confirmation. There are now bound (ex. p4bound32) functions delivering the maximum output buffer size, required only in encoding. This can be used to allocate the output array buffer.