JLD.jl
JLD.jl copied to clipboard
Memory leak when loading multiple arrays in Julia 0.5
trafficstars
Under my configuration, there is a memory leak when loading a JLD file with multiple arrays. I wrote a script which can help test the issue.
The issue is observed when I try to read a JLD file containing multiple arrays, which can be created, for instance, as
N = Int(1e6)
jldopen("randomdata.jld", "w") do io
write(io, "x", rand(N))
write(io, "y", rand(N))
end
Then I load the file several times:
for n = 1:1000
jldopen("randomdata.jld", "r") do io
x = read(io, "x")
y = read(io, "y")
end
end
and I see memory usage constantly increase until the process crashes due to exceeding the limits of system memory.
Some comments:
- this only seems to happen under Julia 0.5 (from Fedora 24 repos). I tried with Julia 0.4 (binaries from http://julialang.org/downloads) and I don't have this problem.
- HDF5 version is 1.8.16 (from Fedora 24 repos).
- when I try using h5open instead of jldopen, I don't see a memory leak:
using HDF5
for n = 1:1000
h5open("randomdata.jld", "r") do io
x = read(io, "x")
y = read(io, "y")
end
end
- apparently there's no memory leak when the file contains only one array, as in
save("randomdata.jld", "x", rand(N)).