Array{Array{T, N}, 1} saved as Array{Any, 1}?

[luiarthur]

First, thanks for this awesome package!

I was wondering if the following is expected behavior:

using BSON
z = [ones(2, 3) for i in 1:2]
typeof(z) # Array{Array{Float64,2},1}

BSON.bson("test.bson", z=z)
d = BSON.load("test.bson")
typeof(d[:z]) # Array{Any,1}

I would have expected typeof(d[:z]) == typeof(z). But that is not the case. This seems to happen when typeof(z) is an array of arrays. But it doesn't happen with arrays. i.e.,

a = ones(3)
BSON.bson("a.bson", a=a)
d2 = BSON.load("a.bson")
typeof(d2[:a]) # Array{Float64, 1}

returns Array{Float64, 1} as expected.

I came across #26, but that seems like a different issue.

Thanks.

[Datseris]

Yeah I confirm and this is a BIG problem for multiple dispatch.

[Datseris]

@MikeInnes do you have any clue of how can this be fixed? Any input would be welcomed here because this issue means that you can't use multiple dispatch on files containing vectors of arrays loaded with BSON.

[DilumAluthge]

This is not just an issue when saving an array of arrays. The same problem happens when saving an array of strings. See https://github.com/MikeInnes/BSON.jl/issues/48

[DilumAluthge]

Here is some code that recursively fixes the types of array.

function _assigned(a::AbstractArray{T, N})::BitArray{N} where T where N
    result::BitArray{N} = BitArray{N}(undef, size(a))
    for i = 1:length(a)
        result[i] = isassigned(a, i)
    end
    return result
end

_correct_typeof(x::T) where T <: Any = T

_correct_typeof(a::A) where A <: AbstractArray{T, N} where T where N = Array{_correct_eltype(a), N}

_correct_eltype(a::A) where A <: AbstractArray{T, N} where T where N = _bottom_to_any(typejoin(_correct_typeof.(a[_assigned(a)])...))

_bottom_to_any(x) = x
_bottom_to_any(::Core.TypeofBottom) = Any

fixtype(x) = x
function fixtype(a::A) where A <: AbstractArray{T, N} where T where N
    S = _correct_typeof(a)
    new_array::S = S(undef, size(a))
    for i = 1:length(a)
        if isassigned(a, i)
            new_array[i] = fixtype(a[i])
        end
    end
    return new_array
end

[DilumAluthge]

Usage

fixtype(a)

Example 1

julia> a = Vector{Any}(undef, 5)
5-element Array{Any,1}:
 #undef
 #undef
 #undef
 #undef
 #undef

julia> a[1] = 1.0
1.0

julia> b = Vector{Any}(undef, 5)
5-element Array{Any,1}:
 #undef
 #undef
 #undef
 #undef
 #undef

julia> b[3] = 3.0
3.0

julia> c = Vector{Any}(undef, 5)
5-element Array{Any,1}:
 #undef
 #undef
 #undef
 #undef
 #undef

julia> c[5] = 5.0
5.0

julia> d = Any[a,b,c]
3-element Array{Any,1}:
 Any[1.0, #undef, #undef, #undef, #undef]
 Any[#undef, #undef, 3.0, #undef, #undef]
 Any[#undef, #undef, #undef, #undef, 5.0]

julia> typeof(d)
Array{Any,1}

julia> e = fixtype(d)
3-element Array{Array{Float64,1},1}:
 [1.0, 4.94066e-324, 2.31974e-314, 9.88131e-324, 4.94066e-324]
 [0.0, 4.94066e-324, 3.0, 9.88131e-324, 4.94066e-324]
 [2.34169e-314, 2.34169e-314, 2.34169e-314, 2.34169e-314, 5.0]

julia> typeof(e)
Array{Array{Float64,1},1}

Example 2

julia> f = Vector{Any}(undef, 5)
5-element Array{Any,1}:
 #undef
 #undef
 #undef
 #undef
 #undef

julia> g = Vector{Any}(undef, 5)
5-element Array{Any,1}:
 #undef
 #undef
 #undef
 #undef
 #undef

julia> f[1] = "hello"
"hello"

julia> g[4] = "world"
"world"

julia> h = Any[a, b]
2-element Array{Any,1}:
 Any[1.0, #undef, #undef, #undef, #undef]
 Any[#undef, #undef, 3.0, #undef, #undef]

julia> typeof(h)
Array{Any,1}

julia> i = fixtype(h)
2-element Array{Array{Float64,1},1}:
 [1.0, 2.31975e-314, 2.3326e-314, 2.33223e-314, 2.24855e-314]
 [2.26145e-314, 2.28997e-314, 3.0, 2.32793e-314, 2.24857e-314]

julia> typeof(i)
Array{Array{Float64,1},1}

[DilumAluthge]

My recommendation is to call fixtype(...) on every Array you load from BSON. This will make sure that you have the correct types and eltypes.

[Datseris]

Yeah but it would also be cool if @MikeInnes gave some insight on how to fix the problem on the BSON level...