TurboPFor-Integer-Compression
TurboPFor-Integer-Compression copied to clipboard
Published
20 hours ago •
powturbo
powturbo
Benchmark: TurboPFor Integer Compression - Skylake i7-6700 3.4GHz
TurboPFor: IcApp Integer Compression Benchmark - Skylake i7-6700 3.4GHz
docs: Document Ids
./icapp test_collection.docs
file: max bits histogram: 00: 0.002% 01:## 1.7% 02:# 0.7% 03: 0.4% 04: 0.2% 05: 0.2% 06:# 0.5% 07:# 1.4% 08:## 1.6% 09:#### 3.6% 10:###### 6.4% 11:######## 8.1% 12:################# 17% 13:############################################# 45% 14:############# 13%
E MB/s size ratio D MB/s function (integer size=32 bits) BOLD=Pareto
1038.82 3017800 21.93% 2661.63 vszenc32 TurboVSimple zigzag
1139.82 3147043 22.87% 4950.83 p4nzenc128v32 TurboPForV zigzag
1076.60 3147043 22.87% 3642.05 p4nzenc32 TurboPFor zigzag
964.09 3186374 23.15% 2922.15 p4nzzenc128v32 TurboPFor zzag/delta
887.67 3190622 23.18% 1954.18 bitshuffleZ+lz Transpose+zigzag+lz
1088.78 3228329 23.46% 2693.93 tpnibbleZ+lz Transpose+zigzag+lz
1122.80 3239377 23.54% 5056.32 p4nd1enc128v32 TurboPForV delta1
1075.09 3239377 23.54% 3680.03 p4nd1enc32 TurboPFor delta1
1331.20 3246518 23.59% 6489.07 p4nzenc256v32 TurboPFor256 zigzag
1034.37 3277695 23.81% 2621.08 tpbyteZ+lz Transpose+zigzag+lz
1135.96 3357179 24.39% 5334.62 p4ndenc128v32 TurboPForV delta
1075.59 3357179 24.39% 3644.94 p4ndenc32 TurboPFor delta
1305.82 3423417 24.87% 6158.08 p4nd1enc256v32 TurboPFor256 delta1
1346.84 3515568 25.54% 1550.80 vbddenc32 TurboVByte zzag delt
1311.54 3528896 25.64% 6330.87 p4ndenc256v32 TurboPFor256 delta
920.32 3531916 25.66% 1058.47 bvzzenc32 bitio zigzag/delta
1094.85 3673776 26.69% 3244.53 tpnibble+lz Transpose+lz
1079.98 3676707 26.71% 3193.34 tpbyte+lz Transpose+lz
981.41 3701682 26.90% 2531.42 tpnibbleX+lz Transpose+xor+lz
983.37 3738695 27.16% 2506.98 tpbyteX+lz Transpose+xor+lz
8699.94 3925193 28.52% 11144.38 v8nd1enc128v32 TByte+TPackV delta1
9571.15 3945921 28.67% 11644.09 v8ndenc128v32 TByte+TPackV delta
2644.76 3948726 28.69% 2783.84 vbzenc32 TurboVByte zigzag
7443.65 4014799 29.17% 9577.81 v8nzenc128v32 TByte+TPackV zigzag
11010.65 4173952 30.33% 12030.87 v8ndenc256v32 TByte+TPackV delta
10286.48 4161012 30.23% 11723.43 v8nd1enc256v32 TByte+TPackV delta1
3054.44 4180490 30.37% 3489.68 vbd1enc32 TurboVByte delta1
3230.82 4181163 30.38% 3716.80 vbdenc32 TurboVByte delta
8755.29 4215033 30.63% 10769.41 v8nzenc256v32 TByte+TPackV zigzag
867.36 4228629 30.72% 1893.95 bitshuffleX+lz Transpose+xor+lz
970.41 4260129 30.95% 2366.86 bitshuffle+lz Transpose+lz
1062.80 4329918 31.46% 1271.79 bvzenc32 bitio zigzag
8738.61 4406850 32.02% 10031.57 bitnzpack128v32 TurboPackV zigzag
4255.82 4406850 32.02% 5017.61 bitnzpack32 TurboPack zigzag
11063.76 4724718 34.33% 10735.81 v8zenc32 TurboByte zigzag
3235.38 4724718 34.33% 4183.38 streamvbyte zzag StreamVByte zigzag
11926.61 4819041 35.01% 11644.09 bitnzpack256v32 TurboPack256 zigzag
12041.39 4889190 35.52% 12223.19 v8d1enc32 TurboByte delta1
12661.74 4889479 35.53% 12850.90 v8denc32 TurboByte delta
10490.33 4889479 35.53% 14101.75 streamvbyte delt StreamVByte delta
385.16 5050551 36.70% 423.20 SPDP SPDP Floating Point
1931.69 5265005 38.25% 8086.55 fpxenc32 TurboFloat XOR
1350.93 5265005 38.25% 1524.01 fpfcmenc32 TurboFloat FCM
997.49 5310026 38.58% 1184.65 fpgenc32 bitio TurboGorilla
1657.63 5552401 40.34% 11926.61 p4nenc128v32 TurboPForV
1578.90 5552401 40.34% 7583.09 p4nenc32 TurboPFor
1883.32 5626136 40.88% 7616.66 FastPFor FastPFor
45.75 5626576 40.88% 4884.07 SimdOptPFor FastPFor SIMD
2247.07 5626832 40.88% 13818.59 SimdFastPFor FastPFor SIMD
1903.38 5638279 40.97% 13045.79 p4nenc256v32 TurboPFor256
8291.15 5653941 41.08% 15343.71 v8nenc128v32 TurboByte+TbPackV
18400.15 5661003 41.13% 15275.60 bitnpack128v32 TurboPackV
10458.44 5661003 41.13% 10263.47 bitnpack32 TurboPack
9331.06 5745570 41.75% 15429.72 v8nenc256v32 TurboByte+TbPackV
19115.71 5748842 41.77% 15464.40 bitnpack256v32 TurboPack256
2062.54 6226583 45.24% 6671.50 vsenc32 TurboVSimple
777.85 6577162 47.79% 1149.34 fpdfcmenc32 TurboFloat DFCM
784.99 6658332 48.38% 1150.49 fp2dfcmenc32 TurboFloat DFCM 2D
2628.09 6682141 48.55% 4124.46 vbenc32 TurboVByte scalar
3850.95 6705452 48.72% 6507.48 maskeydvbyte MasedVByte SIMD
10949.33 6790511 49.34% 11527.06 bitnd1pack128v32 TurboPackV delta1
6674.74 6790511 49.34% 8212.00 bitnd1pack32 TurboPack delta1
12169.15 6816223 49.52% 11854.70 bitndpack128v32 TurboPackV delta
7004.23 6816223 49.52% 9001.51 bitndpack32 TurboPack delta
11498.17 7511530 54.58% 14442.09 v8enc32 TurboByte SIMD
10769.41 7511530 54.58% 14321.86 streamvbyte StreamVByte SIMD
13681.22 8062094 58.58% 10710.75 bitnd1pack256v32 TurboPack256 delta1
14262.50 8076334 58.68% 10854.35 bitndpack256v32 TurboPack256 delta
475.40 9044107 65.71% 2907.33 lz lz
1427.58 10833105 78.71% 5124.09 srlez32 TurboRLE32 ESC zzag
1681.94 13712109 99.63% 5869.22 srlex32 TurboRLE32 ESC xor
1869.00 13763312 100.00% 17532.88 srle32 TurboRLE32 ESC
298.29 13763312 100.00% 17532.88 trle TurboRLE
216.18 13763312 100.00% 17532.88 trlex TurboRLE xor
200.00 13763312 100.00% 17532.88 trlez TurboRLE zigzag
17334.15 13763312 100.00% 17510.58 memcpy memcpy
9151.14 13763312 100.00% 11595.04 tpenc Byte transpose
8891.03 13763312 100.00% 11054.87 tp4enc Nibble transpose
4135.61 13763312 100.00% 4112.13 bitshuffle Bit transpose
* : external library
freqs: Term Frequencies
./icapp test_collection.freqs
file: max bits histogram: 00: 0.001% 01:########################### 27% 02:############ 12% 03:###### 6.0% 04:### 3.1% 05:## 1.6% 06:# 1.0% 07:# 1.0% 08:# 0.9% 09:## 1.9% 10:### 3.2% 11:#### 4.1% 12:######## 8.5% 13:####################### 23% 14:###### 6.5%
E MB/s size ratio D MB/s function (integer size=32 bits) 91.61 1692992 12.30% 4615.46 SimdOptPFor FastPFor SIMD 1204.45 1722849 12.52% 4061.17 vsenc32 TurboVSimple 1402.56 1730336 12.57% 8397.38 p4nenc128v32 TurboPForV 1339.76 1730336 12.57% 4979.49 p4nenc32 TurboPFor 1343.42 1778796 12.92% 6106.17 FastPFor FastPFor 1447.85 1779168 12.93% 12169.15 SimdFastPFor FastPFor SIMD 1662.23 1780534 12.94% 12603.76 p4nenc256v32 TurboPFor256 1701.06 1907713 13.86% 3908.92 tpnibble+lz Transpose+lz 1084.24 1922544 13.97% 2792.88 vszenc32 TurboVSimple zigzag 1435.32 1955549 14.21% 2904.88 tpnibbleX+lz Transpose+xor+lz 1023.67 1958057 14.23% 2334.74 bitshuffleX+lz Transpose+xor+lz 1194.83 1962095 14.26% 3022.91 bitshuffle+lz Transpose+lz 1185.37 1996059 14.50% 5303.78 p4nzenc128v32 TurboPForV zigzag 1126.29 1996059 14.50% 3584.19 p4nzenc32 TurboPFor zigzag 1364.32 2087485 15.17% 7105.47 p4nzenc256v32 TurboPFor256 zigzag 1081.60 2130744 15.48% 4695.77 fpxenc32 TurboFloat XOR 877.76 2130744 15.48% 1330.30 fpfcmenc32 TurboFloat FCM 1326.33 2145784 15.59% 2916.57 tpnibbleZ+lz Transpose+zigzag+lz 1293.42 2219153 16.12% 3770.77 tpbyte+lz Transpose+lz 1015.97 2259515 16.42% 1055.63 bvzenc32 bitio zigzag 1003.67 2300817 16.72% 2364.83 bitshuffleZ+lz Transpose+zigzag+lz 1128.69 2381528 17.30% 2856.05 tpbyteX+lz Transpose+xor+lz 1021.17 2398296 17.43% 2984.24 p4nzzenc128v32 TurboPFor zzag/delta 1137.28 2407925 17.50% 2936.48 tpbyteZ+lz Transpose+zigzag+lz 1004.84 2764046 20.08% 1203.82 bvzzenc32 bitio zigzag/delta 16483.00 2784387 20.23% 17312.33 v8nenc128v32 TurboByte+TbPackV 17488.32 2804702 20.38% 17118.54 bitnpack128v32 TurboPackV 9268.22 2804702 20.38% 10522.40 bitnpack32 TurboPack 1670.91 2883341 20.95% 1931.15 vbddenc32 TurboVByte zazg delt 1110.66 3029119 22.01% 1114.80 fpgenc32 bitio TurboGorilla 8271.22 3146929 22.86% 10202.60 v8nzenc128v32 TByte+TPackV zigzag 20948.71 3150687 22.89% 16764.07 v8nenc256v32 TurboByte+TbPackV 22824.72 3189005 23.17% 16764.07 bitnpack256v32 TurboPack256 8591.33 3189535 23.17% 10340.57 bitnzpack128v32 TurboPackV zigzag 4135.61 3189535 23.17% 5150.94 bitnzpack32 TurboPack zigzag 11546.40 3451880 25.08% 11365.24 vbenc32 TurboVByte scalar 8661.61 3457754 25.12% 16846.15 maskeydvbyte MasedVByte SIMD 5340.82 3475286 25.25% 5265.23 vbzenc32 TurboVByte zigzag 10611.65 3502875 25.45% 12959.80 v8nzenc256v32 TByte+TPackV zigzag 11733.42 3589870 26.08% 13349.47 bitnzpack256v32 TurboPack256 zigzag 810.18 3925934 28.52% 3365.11 lz lz 14087.31 4307847 31.30% 16562.34 v8enc32 TurboByte SIMD 11517.41 4307847 31.30% 16825.56 streamvbyte StreamVByte SIMD 11099.44 4323652 31.41% 10744.19 v8zenc32 TurboByte zigzag 4017.31 4323652 31.41% 4233.56 streamvbyte zzag StreamVByte zigzag 1048.79 5742202 41.72% 5785.33 p4ndenc128v32 TurboPForV delta 1001.91 5742202 41.72% 3535.40 p4ndenc32 TurboPFor delta 1258.30 5773355 41.95% 7098.15 p4ndenc256v32 TurboPFor256 delta 11793.75 7227089 52.51% 12432.98 v8denc32 TurboByte delta 10271.12 7227089 52.51% 12062.49 streamvbyte delt StreamVByte delta 8705.44 7231250 52.54% 11703.49 v8ndenc256v32 TByte+TPackV delta 8077.06 7234307 52.56% 10949.33 v8ndenc128v32 TByte+TPackV delta 1408.88 7343213 53.35% 1508.14 vbdenc32 TurboVByte delta 1277.10 10319134 74.98% 5386.81 srle32 TurboRLE32 ESC 735.30 10350497 75.20% 1170.55 fpdfcmenc32 TurboFloat DFCM 731.12 10690707 77.68% 1161.46 fp2dfcmenc32 TurboFloat DFCM 2D 1124.82 10963622 79.66% 4061.17 srlex32 TurboRLE32 ESC xor 853.54 10981857 79.79% 4564.94 p4nd1enc128v32 TurboPForV delta1 840.30 10981857 79.79% 3156.72 p4nd1enc32 TurboPFor delta1 1024.06 11014587 80.03% 5551.96 p4nd1enc256v32 TurboPFor256 delta1 1348.15 11113500 80.75% 4623.21 srlez32 TurboRLE32 ESC zzag 169.10 11621965 84.44% 191.26 SPDP SPDP Floating Point 7451.71 11667495 84.77% 9261.98 v8nd1enc128v32 TByte+TPackV delta1 8029.93 11691692 84.95% 9712.99 v8nd1enc256v32 TByte+TPackV delta1 10016.96 11716644 85.13% 10434.65 v8d1enc32 TurboByte delta1 1137.46 13329244 96.85% 1150.97 vbd1enc32 TurboVByte delta1 11614.60 13727224 99.74% 10458.44 bitndpack128v32 TurboPackV delta 8464.52 13727224 99.74% 9054.81 bitndpack32 TurboPack delta 11527.06 13747047 99.88% 9465.82 bitndpack256v32 TurboPack256 delta 16542.43 13763304 100.00% 16116.28 memcpy memcpy 412.58 13763304 100.00% 16116.28 trle TurboRLE 246.88 13763304 100.00% 16116.28 trlez TurboRLE zigzag 361.16 13763304 100.00% 16097.43 trlex TurboRLE xor 10719.08 13763304 100.00% 11663.82 tpenc Byte transpose 8931.41 13763304 100.00% 11001.84 tp4enc Nibble transpose 4247.93 13763304 100.00% 5147.08 bitshuffle Bit transpose 11693.55 13776743 100.10% 8954.65 bitnd1pack256v32 TurboPack256 delta1 11374.63 13790184 100.20% 10450.50 bitnd1pack128v32 TurboPackV delta1 8251.38 13790184 100.20% 8925.62 bitnd1pack32 TurboPack delta1
Synthetic data: zipfian distribution
./icapp -a1.5 -m0 -M255 -n100M ZIPF
bits histogram: 00:######################################## 40% 01:############## 14% 02:############# 13% 03:########## 10% 04:######## 7.8% 05:###### 5.7% 06:#### 4.1% 07:### 2.9% 08:## 2.1%
E MB/s size ratio D MB/s function (integer size=32 bits) 2368.80 62939886 15.73% 10950.80 p4nenc256v32 TurboPFor256 1327.02 63392759 15.85% 7702.23 p4nenc128v32 TurboPForV 1321.43 63392759 15.85% 4556.85 p4nenc32 TurboPFor 66.39 65060504 16.27% 3077.49 SimdOptPFor FastPFor SIMD 606.96 73459928 18.36% 5349.24 FastPFor FastPFor 631.90 73469416 18.37% 10197.58 SimdFastPFor FastPFor SIMD 1010.83 76345141 19.09% 2866.89 vsenc32 TurboVSimple 1745.84 79163645 19.79% 3332.50 tpnibble+lz Transpose+lz 1464.36 80509600 20.13% 2494.28 tpnibbleX+lz Transpose+xor+lz 1097.09 82870974 20.72% 2541.51 bitshuffle+lz Transpose+lz 975.88 83384370 20.85% 2073.53 bitshuffleX+lz Transpose+xor+lz 1321.55 85243365 21.31% 6370.95 p4nzenc256v32 TurboPFor256 zigzag 1178.72 85546946 21.39% 4991.20 p4nzenc128v32 TurboPForV zigzag 1104.59 85546946 21.39% 3735.66 p4nzenc32 TurboPFor zigzag 1334.30 88589767 22.15% 2936.28 tpbyte+lz Transpose+lz 902.66 89283071 22.32% 2240.92 vszenc32 TurboVSimple zigzag 1010.73 94657651 23.66% 4570.28 fpxenc32 TurboFloat XOR 823.29 94657651 23.66% 1348.86 fpfcmenc32 TurboFloat FCM 1284.95 95923933 23.98% 2332.20 tpbyteX+lz Transpose+xor+lz 1376.65 97552451 24.39% 2321.01 tpnibbleZ+lz Transpose+zigzag+lz 1011.41 98127316 24.53% 2129.66 bitshuffleZ+lz Transpose+zigzag+lz 888.50 98371167 24.59% 926.12 bvzenc32 bitio zigzag 17298.05 99910930 24.98% 12408.10 v8nenc128v32 TurboByte+TbPackV 17356.59 99910930 24.98% 12362.47 bitnpack128v32 TurboPackV 11693.51 99910930 24.98% 10137.62 bitnpack32 TurboPack 17057.57 100332929 25.08% 11994.72 v8nenc256v32 TurboByte+TbPackV 17076.50 100332929 25.08% 11170.06 bitnpack256v32 TurboPack256 1013.86 100628462 25.16% 3083.11 p4nzzenc128v32 TurboPFor zzag/delta 11191.32 101015650 25.25% 10332.45 vbenc32 TurboVByte scalar 6689.75 102074663 25.52% 9524.04 maskeydvbyte MasedVByte SIMD 1161.18 103144618 25.79% 1263.64 fpgenc32 bitio TurboGorilla 1164.71 105990559 26.50% 2202.97 tpbyteZ+lz Transpose+zigzag+lz 3857.24 106284616 26.57% 4009.10 vbzenc32 TurboVByte zigzag 9820.77 112368050 28.09% 10469.01 bitnzpack128v32 TurboPackV zigzag 9686.17 112368050 28.09% 10426.44 v8nzenc128v32 TByte+TPackV zigzag 4189.84 112368050 28.09% 5317.24 bitnzpack32 TurboPack zigzag 12560.05 112825409 28.21% 11054.92 v8nzenc256v32 TByte+TPackV zigzag 12680.30 112825409 28.21% 11036.61 bitnzpack256v32 TurboPack256 zigzag 1650.88 116367689 29.09% 1882.50 vbddenc32 TurboVByte zazg delt 832.48 119294130 29.82% 973.89 bvzzenc32 bitio zigzag/delta 13107.45 125000000 31.25% 12375.09 v8enc32 TurboByte SIMD 11186.62 125000000 31.25% 12123.78 streamvbyte StreamVByte SIMD 10956.20 128705458 32.18% 10696.90 v8zenc32 TurboByte zigzag 3208.47 128705458 32.18% 3850.37 streamvbyte zzag StreamVByte zigzag 656.49 140353625 35.09% 2416.76 lz lz 1002.48 231440944 57.86% 5208.47 p4ndenc128v32 TurboPForV delta 959.54 231440944 57.86% 3770.10 p4ndenc32 TurboPFor delta 1180.67 231486347 57.87% 5985.96 p4ndenc256v32 TurboPFor256 delta 488.03 239203583 59.80% 2701.28 trle TurboRLE 10108.16 245851337 61.46% 10481.08 v8denc32 TurboByte delta 9441.09 245851337 61.46% 10096.17 streamvbyte delt StreamVByte delta 8420.70 246241959 61.56% 10398.52 v8ndenc256v32 TByte+TPackV delta 7895.00 246632587 61.66% 9510.45 v8ndenc128v32 TByte+TPackV delta 1142.53 262025557 65.51% 1198.36 vbdenc32 TurboVByte delta 436.08 263233919 65.81% 1532.21 trlex TurboRLE xor 315.02 263233919 65.81% 1053.77 trlez TurboRLE zigzag 1085.73 291716271 72.93% 5591.59 p4nd1enc256v32 TurboPFor256 delta1 918.79 291995716 73.00% 4686.75 p4nd1enc128v32 TurboPForV delta1 897.86 291995716 73.00% 3595.60 p4nd1enc32 TurboPFor delta1 9353.88 304132355 76.03% 9687.11 v8d1enc32 TurboByte delta1 7881.46 304522974 76.13% 8913.05 v8nd1enc256v32 TByte+TPackV delta1 7441.72 304913602 76.23% 8818.73 v8nd1enc128v32 TByte+TPackV delta1 1026.88 339718007 84.93% 1063.92 vbd1enc32 TurboVByte delta1 168.67 374829697 93.71% 189.64 SPDP SPDP Floating Point 1374.41 384560608 96.14% 9134.30 srle32 TurboRLE32 ESC 709.62 387715234 96.93% 1186.37 fpdfcmenc32 TurboFloat DFCM 710.10 388257392 97.06% 1175.15 fp2dfcmenc32 TurboFloat DFCM 2D 1174.42 392073698 98.02% 5061.88 srlex32 TurboRLE32 ESC xor 1433.31 393849411 98.46% 5286.18 srlez32 TurboRLE32 ESC zzag 14002.66 400000000 100.00% 14043.46 memcpy memcpy 3876.57 400000000 100.00% 3789.60 bitshuffle Bit transpose 8518.79 400000000 100.00% 9106.43 tpenc Byte transpose 8304.44 400000000 100.00% 9598.08 tp4enc Nibble transpose 9278.16 400390622 100.10% 7273.65 bitnd1pack256v32 TurboPack256 delta1 9303.19 400390622 100.10% 7575.18 bitndpack256v32 TurboPack256 delta 7950.55 400781247 100.20% 7644.97 bitnd1pack32 TurboPack delta1 8069.89 400781247 100.20% 7650.09 bitndpack32 TurboPack delta 9284.62 400781247 100.20% 8515.17 bitnd1pack128v32 TurboPackV delta1 9294.98 400781247 100.20% 8564.58 bitndpack128v32 TurboPackV delta
