arrow-julia icon indicating copy to clipboard operation
arrow-julia copied to clipboard

Fix `OffsetArray` support and clean `CategoricalArray` support

Open nalimilan opened this issue 5 months ago • 1 comments
trafficstars

Invalid indices were used with OffsetArrays as 1-based indexing was assumed. Fix this, and always wrap them in a ToArrow objet so that they are consistently turned into 1-based arrays.

This allows dropping special code for CategoricalArray in favor of using the standard DataAPI.refpool API combined with the Arrow extension point added to CategoricalArrays (https://github.com/JuliaData/CategoricalArrays.jl/pull/415).

If this looks good I'll backport https://github.com/JuliaData/CategoricalArrays.jl/pull/415 to a minor CategoricalArrays release, as currently it's only on master (soon to become 1.0), which explains why CI fails. Another interesting option would be to move the Arrow-CategoricalArrays extension to Arrow, which would ensure support works even with older CategoricalArray versions. Let me know what you think.

nalimilan avatar May 29 '25 20:05 nalimilan

Codecov Report

:x: Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 4.90%. Comparing base (3712291) to head (f49dfb0). :warning: Report is 38 commits behind head on main.

Files with missing lines Patch % Lines
src/arraytypes/dictencoding.jl 0.00% 2 Missing :warning:

:exclamation: There is a different number of reports uploaded between BASE (3712291) and HEAD (f49dfb0). Click for more details.

HEAD has 27 uploads less than BASE
Flag BASE (3712291) HEAD (f49dfb0)
35 8
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #556       +/-   ##
==========================================
- Coverage   87.43%   4.90%   -82.54%     
==========================================
  Files          26      26               
  Lines        3288    3305       +17     
==========================================
- Hits         2875     162     -2713     
- Misses        413    3143     +2730     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar May 29 '25 20:05 codecov-commenter