arrow-julia
arrow-julia copied to clipboard
error earlier when number of entries don't match across all fields
right now it throws confusing messages:
julia> Arrow.write(tempname(), df)
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] getindex
@ ./array.jl:924 [inlined]
[2] iterate
@ ~/.julia/packages/Arrow/SFb8h/src/arraytypes/list.jl:171 [inlined]
[3] Arrow.ToList(input::Arrow.ToList{Vector{Bool}, false, Vector{Vector{Bool}}, Int32}; largelists::Bool)
@ Arrow ~/.julia/packages/Arrow/SFb8h/src/arraytypes/list.jl:103
[4] arrowvector(::ArrowTypes.ListKind{false}, x::Arrow.ToList{Vector{Bool}, false, Vector{Vector{Bool}}, Int32}, i::Int64, nl::Int64, fi::Int64, de::Dict{Int64, Any}, ded::Vector{Arrow.DictEncoding}, meta::Nothing; largelists::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{6, Symbol}, NamedTuple{(:dictencode, :maxdepth, :lareglists, :compression, :denseunions, :dictencodenested), Tuple{Bool, Int64, Bool, Nothing, Bool, Bool}}})
Sorry, can you clarify what is going on here? Are you saying you have columns in df (I assume that's a DataFrame?) that have different lengths? i.e. col1 has 10 rows, while col2 has 12 rows or something?
The codepath you're hitting here seems to suggest that a column in your DataFrame is actually a column of Array objects? Are you sure you don't have an element that is something like [#undef] which would result in this error? If you can provide a reproducible example, we can probably fix it really quickly.