Support C data interface

[quinnj]

trafficstars

@sa- and I have started work in #178 and #179

[evetion]

The efforts seem to have stalled (other PR is closed), are there any plans/interest to revive it? I could definitely use this functionality, as GDAL nowadays can expose vector data via an Arrow (C) interface. Benchmarks in other languages have seen significant speedups using it.

[Yuan-Ru-Lin]

Hi I wonder what work needs to be done in #178 in order to close this issue. It seems there have been functionalities to load ArrowSchema and ArrowArray from memory managed by pyarrow already. Do we want to recursively load nested ArrowSchema or ArrowArray?

[ollemartensson]

Hey I'm also in need of this so I did some research and put claude on the task. It has the complete implementation + tensor support. #560

[ollemartensson]

The original pull request has been chopped up into three distinctive PRs and the one that matters for this issue is now self contained:

#561

Changes and updates

• Added a lot of test coverage since the failed CI checks. Property based testing and caught several bugs and actually missing functionality which I think is a testament to how valuable those types of tests can be!
• Updated docs & commit messages & code to follow the ASF Generative Tooling Guidance.

ASF Generative Tooling Guidance Assessment

ASF Generative Tooling Guidance

• [x] Terms and Conditions of Generative AI Tool Claude Code's terms do not place restrictions on use of output that would be inconsistent with the Open Source Definition.
• [x] Output is not copyrightable subject matter
• Technical configuration code, standard API implementations, and test patterns are generally not copyrightable
• Implementation follows established Apache Arrow C Data Interface specification
• Code structure follows standard software engineering patterns
• [x] No third party materials included
• All code generated specifically for this Apache Arrow.jl implementation
• Based on public Apache Arrow specification, not copied from other implementations
• Original technical design and research contributions
• [x] Compatible licensing for any referenced materials
• Apache Arrow C Data Interface specification is publicly available
• All generated code compatible with Apache-2.0 license
• No proprietary or incompatibly licensed content included
• [x] Disclosure and Attribution Requirements
• Each commit includes the required disclosure:

  🤖 Implementation developed with Claude Code assistance Research and Technical Design: Original contribution

• This follows the recommended practice of indicating "tooling used to create the contribution" and includes it "as a token in the source control commit message" as specified in the guidance.
• [x] Contributor Responsibility
• Contributor (user) has reviewed and approved all generated content
• Technical design and research contributions are original work
• Implementation methodology developed under direct human guidance
• All content verified for compliance with Apache-2.0 license terms