JLD.jl icon indicating copy to clipboard operation
JLD.jl copied to clipboard

Loading SharedArray's doesn't redistribute the array across workers

Open azraq27 opened this issue 8 years ago • 0 comments
trafficstars

I just noticed that when I save a SharedArray, the type information is appropriately recreated on load. However, when loaded, the data exists in the primary thread, but is not shared across all the other active workers, as in normally done when creating a SharedArray.

Saving the data:

addprocs(1)
using JLD

data = SharedArray{Float64,1}(rand(100))

# The data exists on the worker thread
fetch(@spawn data)

jldopen("test.jld","w") do f
    write(f,"test",data)
end

Loading the data (make sure it's a different process):

addprocs(1)
using JLD

data = jldopen("test.jld") do f
    read(f,"test")
end

fetch(@spawn data)

The final line there throws this error:

ERROR: LoadError: On worker 2:
Local instance of remote reference not found

Adding data = SharedArray{Float64,1}(data) doesn't fix it, because it's already a SharedArray. Converting it to an Array and then back does fix it, using data = SharedArray{Float64,1}(copy(data)).

Would the best solution be to add a simple conditional that catches SharedArray's, and then reinitializes them through that workaround?

azraq27 avatar Sep 27 '17 17:09 azraq27