Turbo-Range-Coder

Turbo-Range-Coder copied to clipboard

TurboRC: Turbo Range Coder + rANS Asymmetric Numeral Systems

======================================

• Fastest (Branchless) Range Coder / Arithmetic Coder
  • 100% C (C++ headers).
  • OS/Arch: Linux amd/intel, arm, PowerPC, s390x, MacOs+Apple M1. Windows: Mingw, visual c++
  • No other Range Coder / Arithmetic Coder encode or decode faster with better compression
  • Up to 3 times faster than the next fastest range coder with similar compression ratio
  • Can work as bitwise or/and as multisymbol range coder
  • 32 or 64 bits range coder. Big + Little endian
  • Renormalization output 8,16 or 32 bits
  • Easy connection to bit, nibble or byte predictors.
  • Several built-in predictors: simple, dual speed, fsm
  • Built-in order0, order1, order2, Sliding Context, Context mixing,
    - Run Length Encoding, Gamma Coding, Rice Coding,
    - Bit entropy coding,
    - Turbo VLC: novel Variable Length Coding for large integers,
    - MTF (Move-To-Front) / QLFC (Quantized Local Frequency Coding)
    
  • Fast full 8/16 bits BWT: Burrows-Wheeler compression/decompression w/
    • preprocessing : lzp and utf-8
    • postprocessing: QLFC (Quantized Local Frequency Coding),
      RLE and Bit entropy coder
  • BWT :libsais + optimized libdivsufsort + optimized inverse bwt included
  • static + adaptive CDF - cumulative distribution functions
  • stdin/stdout file compressor included
  • TurboRC App for benchmarking all the functions and test allmost all byte, integer, floating point, date and timestamp file types.
    • read and convert text, csv or binary files to 8/16/32 bits before processing
    • set predictor and parameters at the command line
  • Preprocessing:
    • :new: Linear Quantization for 16/32/64 floating point to 8/16/32/64 integer (2023.08)
• Asymmetric Numeral Systems
  • :new: Fastest Adaptive CDF rANS Asymmetric Numeral Systems / SSE, AVX2. (2023.05)
  • :new: bitwise ANS (2023.05)
  • No other adaptive ANS encode or decode faster with better compression

LICENSE

• GPL 3.0
• A commercial license is available. Contact us at powturbo [AT] gmail.com for more information.

Usage examples

    ./turborc -e0   file           " benchmark all basic functions using the default simple predictor
    ./turborc -e20  file           " byte gamma coding + rc
    ./turborc -e1,2 file
    ./turborc -e0 -pss -r47 file   " use dual speed predictor with parameters 4 and 7
    ./turborc -e0 -psf -r1 file    " use FSM predictor with filename "FSM1.txt"
    ./turborc -e0 file -Os         " raw 16 bits input
    ./turborc -e0 file -Ou         " raw 32 bits input
    ./turborc -e0 file -Ft         " text file (one integer/line) 
    ./turborc -e0 file -Fc         " text file with multiple integer entries (separated by non-digits characters ex. 456,32,54)
    ./turborc -e0 file -Fc -v5     " like prev., display the first 100 values read
    ./turborc -e0 file -Fcf        " text file with multiple floating-point entries (separated by non-digits characters ex. 456.56,32.1,54)
    ./turborc -e0 file -Fru -Ob    " convert raw 32 bits input to bytes before processing possibly truncating large values
    ./turborc -e0 file -Ft -K3 -Ou " convert column 3 of a csv text file to 32 bits integers
    ./turborc -e0 file -pss -r47   " benchmark all basic functions using the dual speed predictor with paramters 4 and 7
    ./turborc -e0 file -psf -r1    " benchmark all basic functions using the fsm predictor with the paramter file FSM1.txt

Benchmark

see also Entropy Coder Benchmark

File: enwik8bwt generated BWT (wikipedia XML 100MB )

    > turborc -e0 enwik8bwt
    

BWT Benchmark: TurboRC vs the best BWT compressors (2023.04)

• bsc
• bzip3
• bzip2

- enwik8 - 100.000.000 bytes EN Wikipedia

(bold = pareto) MB=1.000.000

- Silesia - Compression Corpus (211 MB mixed binary + text)

- English.100mb text files from Gutenberg Project

- html8 : 100MB random html pages from Alexa 1m Top sites

- enwik9 - 1GB EN Wikipedia

- test1.txt - 1GB ZH (chineese) Wikipedia from GDCC2021

- Text log file:NASA access log 200MB

File Compression

Range Coder

    ./turborc -1 inputfile outputfile         "order 0 simple
    ./turborc -2 inputfile outputfile         "order 1 simple

Range Coder + RLE

    ./turborc -1 inputfile outputfile         "order 0 simple
    ./turborc -2 inputfile outputfile         "order 1 simple
    ./turborc -d inputfile outputfile          "decompress

BWT (Burrows-Wheeler) + QLFC (Quantized Local Frequency Coding) + TurboRC

    ./turborc -20e# inputfile outputfile -l# [-Os]  "bwt compression 
	           #:0:store, 2:bit ec, 3/4:RLE, 5/6:RLE o1, 7/8:QLFC, 9:Max
    ./turborc -d inputfile outputfile             "decompress

Compile:

    Download or clone TurboRC
	git clone --recursive https://github.com/powturbo/Turbo-Range-Coder.git
	cd Turbo-Range-Coder
    

Linux, MacOS, Windows (MingW), Clang,... (see also makefile)

	make
or
	make AVX2=1                                "compile for recent architectures >= haswell
 

Windows visual c++

	nmake /f makefile.vs

Windows visual studio c++

	project files in directory vs/vs2022

Function usage:

See examples in "turborc.c"

Environment:

OS/Compiler (32 + 64 bits):

• Windows: Visual C++ (2022)
• Windows: MinGW-w64 makefile
• Linux amd/intel: GNU GCC (>=4.6)
• Linux amd/intel: Clang (>=3.2)
• Linux arm: aarch64
• MaxOS: XCode (>=9) + Apple M1
• PowerPC ppc64le
• IBM s390x

References:

• References:
  • Entropy Coder Benchmark

Last update: 06 AUG 2023