need periodic distributed GC for quicker release of remote objects

[bdqp]

Overwriting RemoteRef variables does not release memory.

Start Julia from bash: julia -p 2 Check memory: free -m (used = 7398) In Julia: r = remotecall(2, rand, 17000, 17000); Check memory: free -m (used = 9617)

Now overwrite r:

In Julia: r = remotecall(2, rand, 17000, 17000); Check memory: free -m (used = 11834) In Julia: gc() Check memory: free -m (used = 11814)

I would expect the overwritten r to have its memory released by the garbage collector.

[amitmurthy]

for i in 1:20
     r = remotecall(2, rand, 17000, 17000); 
     @everywhere gc()
end

completed without any issues on 0.3.9

The issue is that a RemoteRef is only the size of 3 ints, and till it is locally garbage collected, it does not send a "delete my reference to the stored value" message on worker 2. Forcing a gc everywhere is one way to ensure collection is done sooner than later - not an ideal solution, though.

[bdqp]

Hi Amit - I guess the point I was trying to make is that the memory used by processor 2 is not released automatically even after the reference to it no longer exists.

As you say, manually calling the gc() is not ideal but julia is not doing any gc for this situation otherwise.

[amitmurthy]

I concur. I was just trying to explain how the distributed gc is implemented. Local gc may not actually collect r if its size is too small. And the distributed gc is dependent on the finalizers on RemoteRef's being run in order for the remote value to be ultimately freed.

[bdqp]

Just a little follow up - sorry I'm learning as I go here :) The command below seems to suggest it will clean up memory.

take!(RemoteRef) Fetch the value of a remote reference, removing it so that the reference is empty again.

However, if I execute the following command several times the memory gets consumed until it runs out. I was expecting the memory consumed by the variable on worker 2 to be cleared.

tic(); tic(); r = remotecall(2, rand, 7000, 7000); toc(); f = take!(r); toc()

[sbromberger]

Possibly related? https://groups.google.com/forum/#!topic/julia-users/0ab8qxTdVIg

[ViralBShah]

@amitmurthy I believe most of these issues are now fixed. Can we close this?

[amitmurthy]

No. While not a "leak", an explicit gc call still helps in an early release of the remote object.