pyahocorasick icon indicating copy to clipboard operation
pyahocorasick copied to clipboard

Streamable pickle

Open WojciechMula opened this issue 5 years ago • 0 comments

An attempt to solve #102.

What was done: instead of creating pickle data before pickling, we create an iterator which was meant to yield small portions of data on demand.

While the approach seemed to be a good solution, it appears that both pickle and _pickle modules collect in memory whole dump (despite the way it's created) and then, in the end, flush everything to the output file.

WojciechMula avatar Dec 15 '18 14:12 WojciechMula