JLD.jl icon indicating copy to clipboard operation
JLD.jl copied to clipboard

Memory leak when loading multiple arrays in Julia 0.5

Open jipolanco opened this issue 9 years ago • 0 comments
trafficstars

Under my configuration, there is a memory leak when loading a JLD file with multiple arrays. I wrote a script which can help test the issue.

The issue is observed when I try to read a JLD file containing multiple arrays, which can be created, for instance, as

N = Int(1e6)
jldopen("randomdata.jld", "w") do io
    write(io, "x", rand(N))
    write(io, "y", rand(N))
end

Then I load the file several times:

for n = 1:1000
    jldopen("randomdata.jld", "r") do io
        x = read(io, "x")
        y = read(io, "y")
    end
end

and I see memory usage constantly increase until the process crashes due to exceeding the limits of system memory.

Some comments:

  • this only seems to happen under Julia 0.5 (from Fedora 24 repos). I tried with Julia 0.4 (binaries from http://julialang.org/downloads) and I don't have this problem.
  • HDF5 version is 1.8.16 (from Fedora 24 repos).
  • when I try using h5open instead of jldopen, I don't see a memory leak:
using HDF5
for n = 1:1000
    h5open("randomdata.jld", "r") do io
        x = read(io, "x")
        y = read(io, "y")
    end
end
  • apparently there's no memory leak when the file contains only one array, as in save("randomdata.jld", "x", rand(N)).

jipolanco avatar Oct 21 '16 07:10 jipolanco