anaconda icon indicating copy to clipboard operation
anaconda copied to clipboard

Memory leak (possible goroutine leak) in throttledQuery

Open carbocation opened this issue 8 years ago • 1 comments

I run a long-lived binary that queries multiple social media providers. I use anaconda to query Twitter. It makes hundreds of queries per minute, and over the course of days/weeks, it ends up using all available RAM on the system.

Profiling, it seems that throttledQuery itself is the source of the leaks. Here is the inuse_space graph from the profile (note: it's an SVG and can be clicked on / zoomed in): profiled image

As you can see, I terminated the program when 300MB had leaked (the rest of the program uses a few dozen MB total). I'm guessing that this is a goroutine leak, but I'm still working on profiling that.

carbocation avatar Nov 30 '17 16:11 carbocation

Memory leaks are anticipated in the docs ( https://github.com/ChimeraCoder/anaconda#query-queue-persistence ) but it seems that rather than coding around them, it might be possible to address the leaks by restructuring the function itself.

carbocation avatar Nov 30 '17 16:11 carbocation