Trouble writing multiple files in a row in the same environnement

[ghost]

trafficstars

I am having a weird error when running exactly the same function that write processed data in a .jld file. I am doing something like this :

func( param ) # No error, readable and correct written file
func( param ) # Error ! ( see below )

where func will load data from a .h5 file, process it and then call save( filename, varname, data) Despite the fact that my function func is fully deterministic, I can't call it multiple times in a row in the same environnement. I can only call it once in a session ( which always succeed with any parameters ). The written data structure is composed of a dictionnary of 2D arrays of histograms and arrays of 2D arrays of histograms ( Histogram type from StatsBase ).

I am using Julia 1.0.1 on Windows 10.

Thank you in advance for any advice

HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dio.c line 225 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dio.c line 347 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dio.c line 783 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dcontig.c line 580 in H5D__contig_write(): contiguous write failed
    major: Dataset
    minor: Write failed
  #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dscatgath.c line 678 in H5D__scatgath_write(): datatype conversion failed
    major: Dataset
    minor: Can't convert datatypes
  #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5T.c line 4819 in H5T_convert(): data type conversion failed
    major: Attribute
    minor: Unable to encode value
  #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Tconv.c line 2571 in H5T__conv_struct_opt(): unable to convert compound datatype member
    major: Datatype
    minor: Unable to initialize object
  #007: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5T.c line 4819 in H5T_convert(): data type conversion failed
    major: Attribute
    minor: Unable to encode value
  #008: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Tconv.c line 2172 in H5T__conv_struct(): not a datatype
    major: Datatype
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dio.c line 225 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dio.c line 347 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dio.c line 783 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dcontig.c line 580 in H5D__contig_write(): contiguous write failed
    major: Dataset
    minor: Write failed
  #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Dscatgath.c line 678 in H5D__scatgath_write(): datatype conversion failed
    major: Dataset
    minor: Can't convert datatypes
  #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5T.c line 4819 in H5T_convert(): data type conversion failed
    major: Attribute
    minor: Unable to encode value
  #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Tconv.c line 2571 in H5T__conv_struct_opt(): unable to convert compound datatype member
    major: Datatype
    minor: Unable to initialize object
  #007: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5T.c line 4819 in H5T_convert(): data type conversion failed
    major: Attribute
    minor: Unable to encode value
  #008: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Tconv.c line 2172 in H5T__conv_struct(): not a datatype
    major: Datatype
    minor: Inappropriate type
ERROR: LoadError: Error writing dataset
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] h5d_write at C:\Users\me\.julia\packages\HDF5\b6QoH\src\HDF5.jl:2133 [inlined]
 [3] h5d_write at C:\Users\me\.julia\packages\HDF5\b6QoH\src\HDF5.jl:1985 [inlined]
 [4] writearray at C:\Users\me\.julia\packages\HDF5\b6QoH\src\HDF5.jl:1872 [inlined]
 [5] #write_compound#24(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::JLD.JldGroup, ::String, ::StatsBase.Histogram{Int64,1,Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:707
 [6] write_compound at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:694 [inlined]
 [7] #_write#23 at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:690 [inlined]
 [8] _write at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:690 [inlined]
 [9] macro expansion at .\printf.jl:159 [inlined]
 [10] write_ref(::JLD.JldFile, ::StatsBase.Histogram{Int64,1,Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:657
 [11] h5convert_array(::JLD.JldFile, ::Array{StatsBase.Histogram,2}, ::JLD.JldDatatype, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:602
 [12] #_write#20(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::JLD.JldGroup, ::String, ::Array{StatsBase.Histogram,2}, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:563
 [13] _write at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:561 [inlined]
 ... (the last 5 lines are repeated 1 more time)
 [19] macro expansion at .\printf.jl:159 [inlined]
 [20] write_ref(::JLD.JldFile, ::Array{Array,1}, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:657
 [21] macro expansion at C:\Users\me\.julia\packages\JLD\1BoSz\src\jld_types.jl:648 [inlined]
 [22] h5convert!(::Ptr{UInt8}, ::JLD.JldFile, ::JLD.AssociativeWrapper{String,Array,Dict{String,Array}}, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\jld_types.jl:664
 [23] #write_compound#24(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::JLD.JldGroup, ::String, ::JLD.AssociativeWrapper{String,Array,Dict{String,Array}}, ::JLD.JldWriteSession) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:700
 [24] write_compound at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:694 [inlined]
 [25] #_write#23 at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:690 [inlined]
 [26] _write at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:690 [inlined]
 ... (the last 8 lines are repeated 1 more time)
 [35] #write#17 at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:514 [inlined]
 [36] write at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:514 [inlined]
 [37] (::getfield(JLD, Symbol("##35#36")){String,Results,Tuple{}})(::JLD.JldFile) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:1223
 [38] #jldopen#14(::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:compatible, :compress),Tuple{Bool,Bool}}}, ::Function, ::getfield(JLD, Symbol("##35#36")){String,Results,Tuple{}}, ::String, ::Vararg{String,N} where N) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:246
 [39] #jldopen at .\none:0 [inlined]
 [40] #save#34(::Bool, ::Bool, ::Function, ::FileIO.File{FileIO.DataFormat{:JLD}}, ::String, ::Results) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:1221
 [41] save(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String, ::Results) at C:\Users\me\.julia\packages\JLD\1BoSz\src\JLD.jl:1218
 [42] #save#15(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String, ::String, ::Vararg{Any,N} where N) at C:\Users\me\.julia\packages\FileIO\lt9I0\src\loadsave.jl:118
 [43] save at C:\Users\me\.julia\packages\FileIO\lt9I0\src\loadsave.jl:118 [inlined]
 [44] save_to_file_jld(::Results, ::String, ::String) at c:\Users\me\Documents\code\analysis\processor.jl:52
 [45] statistic_analysis(::Int64, ::Float64) at c:\Users\me\Documents\code\analysis\analysis.jl:203
 [46] top-level scope at none:0
 [47] include at .\boot.jl:317 [inlined]
 [48] include_relative(::Module, ::String) at .\loading.jl:1041
 [49] include(::Module, ::String) at .\sysimg.jl:29
 [50] exec_options(::Base.JLOptions) at .\client.jl:229
in expression starting at c:\Users\me\Documents\code\analysis\data_processing.jl:29

[ghost]

So I rewrote my loading and saving functions. I had the following environnement and functions :

using JLD
using HDF5

function load_data_h5( path, varname)
    data = h5read( path, varname )
    return data
end

function save_data_jld( data, path, varname)
     save( path ,varname ,data )
end

With this environnement I was having the error above.

I then changed it to

using JLD
using HDF5

function load_data_h5( path, varname)
    file = h5open( path, "r" )
    data = read( file ,varname )
    close( file )
end

function save_data_jld( data, path, varname)
     file=jldopen(path, "w")
     write(file, varname, data)
     close(file)
end

And now everything works fine. Hope this will help !