Distributed.jl icon indicating copy to clipboard operation
Distributed.jl copied to clipboard

Large number of temporary shared arrays crashes Julia. Need a more aggressive gc?

Open thraen opened this issue 8 years ago • 18 comments

On my computer julia always crashes on this code:

@everywhere const m     = 100
@everywhere const n     = 100
@everywhere const T     = 100

@everywhere @inline function slice!(x, t, p)
    #   do something
end

function shared_test(p)
    x   = SharedArray(Float64, (m, n, T), init= S -> S[localindexes(S)] = 0.0)

    @sync @parallel for t= 1:T
        slice!(x, t, p)
    end
    return x
end 

function doit()
    p   = rand(m,n,T)

    for i=1: 10000000000
        x = shared_test(p)
        println(i)
    end
end

doit()

This triggers a signal (7): Bus error which results in the workers beeing killed.

I tested this, because I wanted to isolate a memory leak that I suspect. I think it also occurs here, but Julia crashes before I can tell.

I think, if I leave out the Initialization:

x   = SharedArray(Float64, (m, n, T), init= S -> S[localindexes(S)] = 0.0)

by changing the line to

x   = SharedArray(Float64, (m, n, T))

there is no memory leak, but then there are other errors. I am also not sure if the parallel loop is relevant for this.

versioninfo:

Julia Version 0.4.4-pre+35
Commit 1a9429e (2016-02-12 13:21 UTC)
Platform Info:
  System: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Phenom(tm) II X4 955 Processor
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Barcelona)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

thraen avatar Mar 09 '16 03:03 thraen