TurboPFor-Integer-Compression
TurboPFor-Integer-Compression copied to clipboard
powturbo
Fastest Integer Compression
File with ascending integers from -128 to 127 incrementing by 1 `icapp foo.txt -Ftb -e65` > ERROR at 127:in=ff,0,1,2 dec=c0,80,0,8 cmp=2 If I remove one negative integer from the file...
[Quantile Compression/PCodec](https://github.com/mwlon/quantile-compression) is claiming 35%-71% better compression than zstd. I've integrated the rust library into TurboPFor using the ffi bindings for comparison purpose. We use the synthetic [dataset](https://github.com/mwlon/pcodec/tree/main/quantile-compression/q_compress/assets) provided in...
Most of time series databases (ex. DuckDB) are storing floating point data as 64 bits. They are reporting some [extraordinary](https://github.com/panagiotisl/chimp/issues/2) compression ratio by using a gorilla/chimp like algorithm. However as...
> icapp ZIPF zipf alpha=1.50 range[0..255].n=1000000 bits histogram: 00:######################################## 40% 01:############## 14% 02:############# 13% 03:########## 10% 04:######## 7.7% 05:###### 5.7% 06:#### 4.1% 07:### 2.9% 08:## 2.1% size ratio E MB/s...
- 2D/3D datasets: +[Float Compression dataset](https://github.com/aras-p/float_compr_tester/tree/main/data) : [description](https://aras-p.info/blog/2023/01/29/Float-Compression-0-Intro/) +[Datasets for Benchmarking Floating-Point Compressors](https://dps.uibk.ac.at/~fabian/datasets/) - Lossless floating point compression: icapp sq1024x1024x4.f32 -R0 -Ff -I15 -J15 -e105,143,80,102 -Ezstd,15 option -R0 = automatic...
TurboBitByte (=TurboByte+TurboPack) compress considerably better and can be more than 3 times faster than streamvbyte. icapp ZIPF -I15 -J15 -v0 -e45,50,55,130 zipf alpha=1.50 range[0..255].n=1000000 E MB/s size ratio D MB/s...
Quantization of the floating point mantissa by zeroing the rightmost bits according to a maximum allowed pointwise relative error. - Floating point datasets: +[Float Compression dataset](https://github.com/aras-p/float_compr_tester/tree/main/data) : [description](https://aras-p.info/blog/2023/01/29/Float-Compression-0-Intro/) +[Datasets for...
when input buffer is less than 8 bytes these will corrupt memory. I think these start writing at tmp[-4]. If input buffer size is 0, then lz4 will be called...
Are there any guarantees about the max output buffer required for Turbo PFor functions? I haven't seen any in the docs, and I tried to get an idea by skimming...
I wanted to report that I am not able to build the library with GCC 14. It seems that this version turned some warnings into errors (see [https://gcc.gnu.org/gcc-14/porting_to.html](https://gcc.gnu.org/gcc-14/porting_to.html)). As far...
