closed streams leaking with stream/transform

[mccraigmccraig]

i concatenated vectors from a stream using (stream/transform (mapcat identity) s)

this seemed to cause a leak as described here: https://github.com/mpenet/alia/pull/115

investigation showed lots of our EntityInstance records from db queries referenced from the downstream graph metadata in manifold.stream.graph/handle->downstreams - however the system making the queries was running ok, and since it mostly reduces over streams to get results, it seems that streams were being closed yet somehow remaining in the handle->downstreams map

i'm inferring that it's the stream/transform causing the problem because replacing it with a simple connect-via got rid of the memory leak

i've tried to repro this problem in isolation, and so far failed to do so, but i'm creating this issue in case anyone else has been observing similar behaviour

[KingMob]

I'm busy with a RL move, but I'll take a look in May

[KingMob]

@mccraigmccraig I'm taking a deeper look at this, and I made a test gist to see if I could replicate it, based on what's written in the issue. I highly doubt it replicates your conditions, but at least under Manifold 0.2.3, with an infinite seq of small vecs, it's not running out of memory according to YourKit. How are you closing streams, and what are they doing beforehand? The

I looked into the use of handle->downstreams, and it appears to be a concurrent version of a WeakHashMap, implemented with ConcurrentHashMap. All the keys are WeakReferences, so they shouldn't in theory prevent anything from being GCed, but I noticed that all the WeakReference keys are refs of the streams themselves. I'm not sure, but this might be a problem, since if there's a cycle of normal strong references anywhere from a stream back to itself, nothing will get automatically discarded from handle->downstreams. There seem to be a lot of manual calls to .remove streams from handle->downstreams that should sufficient, but I haven't checked it all yet to make sure it's OK.

Is there any more info you can share? Is it still a problem in Manifold 0.2.3? Do you have a memory profile recording I could look at? Anything would help.