ziplm
ziplm copied to clipboard
Use zlib through CFFI like a real cowboy
Reinitializing the compressor on each model evaluation is wasteful. Use the raw power of zlib's advanced API. There are two things of note here.
deflateCopy
You can make a deflate stream for each word in the alphabet plus the training data. Then, to evaluate a prefix, you copy that stream with deflateCopy
and just append the prefix. So it's a suffix, really, if you think about it.
deflateSetDictionary
Wow, the raw power. You can set the dictionary that zlib uses.