memory leak in 1.4.0

[WojciechMula]

Hi, i'm sorry for opening such an old issue, but i'm currently experiencing the same issue. I'm using version 1.4.0 now and getting small steady memory leaks (after debugging with tracemalloc) on:

A = ahocorasick.Automaton() MyList = [...] for x in MyList: A.add_word(y, (y, z))

is there a chance this bug has returned

Thanks, Eden.

Originally posted by @EdenAzulay in https://github.com/WojciechMula/pyahocorasick/issues/81#issuecomment-705534956

[AlonSh]

I think that I'm also experiencing the same memory leak on add_word. Would love to see any updates :)

Edit - I was experiencing a different memory leak. My leak originated from using multiprocessing Pool and some issue with passing ahocorasick automaton between workers, I think there's some issue with serialization causing old objects not to be cleaned.

[WojciechMula]

@AlonSh could you please provide some minimal example?

[AlonSh]

Yeah: create some automaton create a multiprocessing Pool and do:

pool.apply_async(
            run_automaton,
            (automaton, text),
            callback=callback_success,
            error_callback=_my_error_callback,
        )

and you'll see your memory exploding after some calls.

[WojciechMula]

Great! Thank you.

[pombredanne]

I am pushing tests to run on the CI on many Linux ... but while I can have it fail locally on Ubuntu 16... the tests seem to pass on more recent linux. I wonder if this is not dependent on a certain version of the compiler? Otherwise, this is a head scratcher. @AlonSh FWIW, I recycle processes after a 1000 calls in my pools to cope with leaks. Not perfect, but a workaround at least. See for instance https://github.com/nexB/scancode-toolkit/blob/e080f8354bed5813df9b619efe575ce9931a5a5b/src/scancode/cli.py#L1209

[Azzonith]

Hello guys,

Is there any update on this issue? I've tested a library version 2.0.0 today and memory consumption added up every time automaton was used in ProcessPoolExecutor futures. We had to stop the service after RAM consumption crossed 140GB. I attempted to build an image FROM ubuntu:20.04, python:3.8, python:3.10. The issue is reproduced every time. The latest usable lib version for us remains 1.1.8. Please let me know if there is any troubleshooting info I could provide for the research.