ziplm icon indicating copy to clipboard operation
ziplm copied to clipboard

Use zlib through CFFI like a real cowboy

Open lericson opened this issue 1 year ago • 0 comments

Reinitializing the compressor on each model evaluation is wasteful. Use the raw power of zlib's advanced API. There are two things of note here.

deflateCopy

You can make a deflate stream for each word in the alphabet plus the training data. Then, to evaluate a prefix, you copy that stream with deflateCopy and just append the prefix. So it's a suffix, really, if you think about it.

deflateSetDictionary

Wow, the raw power. You can set the dictionary that zlib uses.

lericson avatar Jul 16 '23 08:07 lericson