arrow-julia icon indicating copy to clipboard operation
arrow-julia copied to clipboard

error earlier when number of entries don't match across all fields

Open Moelf opened this issue 3 years ago • 1 comments

right now it throws confusing messages:

julia> Arrow.write(tempname(), df)
ERROR: UndefRefError: access to undefined reference
Stacktrace:
  [1] getindex
    @ ./array.jl:924 [inlined]
  [2] iterate
    @ ~/.julia/packages/Arrow/SFb8h/src/arraytypes/list.jl:171 [inlined]
  [3] Arrow.ToList(input::Arrow.ToList{Vector{Bool}, false, Vector{Vector{Bool}}, Int32}; largelists::Bool)
    @ Arrow ~/.julia/packages/Arrow/SFb8h/src/arraytypes/list.jl:103
  [4] arrowvector(::ArrowTypes.ListKind{false}, x::Arrow.ToList{Vector{Bool}, false, Vector{Vector{Bool}}, Int32}, i::Int64, nl::Int64, fi::Int64, de::Dict{Int64, Any}, ded::Vector{Arrow.DictEncoding}, meta::Nothing; largelists::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{6, Symbol}, NamedTuple{(:dictencode, :maxdepth, :lareglists, :compression, :denseunions, :dictencodenested), Tuple{Bool, Int64, Bool, Nothing, Bool, Bool}}})

Moelf avatar Oct 11 '22 23:10 Moelf

Sorry, can you clarify what is going on here? Are you saying you have columns in df (I assume that's a DataFrame?) that have different lengths? i.e. col1 has 10 rows, while col2 has 12 rows or something?

The codepath you're hitting here seems to suggest that a column in your DataFrame is actually a column of Array objects? Are you sure you don't have an element that is something like [#undef] which would result in this error? If you can provide a reproducible example, we can probably fix it really quickly.

quinnj avatar Oct 12 '22 22:10 quinnj