HDF5.jl

HDF5.jl copied to clipboard

@everywhere using HDF5
@everywhere h5 = h5open("test.h5","w")
for j=1:10 
    g_create(h5,"$j")
    h5["$j/j"]=j
end
map(j->read(h5["$j/j"]), names(h5))
pmap(j->read(h5["$j/j"]), names(h5)) # this line craps out when julia is started with 'julia -p 3'

Is this something I should be able to get working? By default on OSX Homebrew installs the non-threadsafe version of HDF5 as far as I can tell. I tried to override with Homebrew.rm("HDF5") followed by ./brew install --force-bottle staticfloat/juliadeps/hdf5 --enable-threadsafe. It didn't fix it or even change the error that I get.

Julia is not presently multithreaded. Those are separate processes. HDF5 does not support this for writing (writing from multiple processes simultaneously would probably corrupt it), but for reading you could open the file separately in each process.

Ok thanks. I'm going to have to do more than just add a p to my code.

We could think about overloading Base.serialize for HDF5File/JLDFile, though, so that sending them to another process would automatically open the file there (or maybe throw an error if the file is opened for writing).