Distributed.jl
Distributed.jl copied to clipboard
Large number of temporary shared arrays crashes Julia. Need a more aggressive gc?
On my computer julia always crashes on this code:
@everywhere const m = 100
@everywhere const n = 100
@everywhere const T = 100
@everywhere @inline function slice!(x, t, p)
# do something
end
function shared_test(p)
x = SharedArray(Float64, (m, n, T), init= S -> S[localindexes(S)] = 0.0)
@sync @parallel for t= 1:T
slice!(x, t, p)
end
return x
end
function doit()
p = rand(m,n,T)
for i=1: 10000000000
x = shared_test(p)
println(i)
end
end
doit()
This triggers a signal (7): Bus error
which results in the workers beeing killed.
I tested this, because I wanted to isolate a memory leak that I suspect. I think it also occurs here, but Julia crashes before I can tell.
I think, if I leave out the Initialization:
x = SharedArray(Float64, (m, n, T), init= S -> S[localindexes(S)] = 0.0)
by changing the line to
x = SharedArray(Float64, (m, n, T))
there is no memory leak, but then there are other errors. I am also not sure if the parallel loop is relevant for this.
versioninfo:
Julia Version 0.4.4-pre+35
Commit 1a9429e (2016-02-12 13:21 UTC)
Platform Info:
System: Linux (x86_64-pc-linux-gnu)
CPU: AMD Phenom(tm) II X4 955 Processor
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Barcelona)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3