arrow-julia icon indicating copy to clipboard operation
arrow-julia copied to clipboard

Support DataAPI.jl metadata API

Open bkamins opened this issue 3 years ago • 3 comments

Hi @quinnj - could you please add to the release plan of Arrow.jl support for https://github.com/JuliaData/DataAPI.jl/pull/48 for the created Arrow tables. Only read methods need to be implemented for Arrow tables:

  • DataAPI.metadata
  • DataAPI.metadatakeys
  • DataAPI.colmetadata
  • DataAPI.colmetadatakeys

bkamins avatar Sep 20 '22 08:09 bkamins

@quinnj - we should get back to it as the issue got forgotten (at least by me). In particular we should also add an automatic writing of metadata from objects that support metadata (instead of using metadata and colmetadata keywords in writer).

Do you have any comments on this (or I should just propose a PR)?

bkamins avatar Aug 03 '23 11:08 bkamins

Ah yes, just make a PR; I think it should be pretty straightforward.

quinnj avatar Aug 03 '23 14:08 quinnj

In https://github.com/apache/arrow-julia/pull/481 I have implemented it for Arrow.Table.

My follow-up questions are:

  • I was not sure how Arrow.Stream should be handled for metadata reading.
  • I was not sure for Arrow.write and Arrow.Writer and Arrow.append how to handle writing metadata. In particular, if one writes to an already existing Arrow file what should be allowed (i.e. if adding/updating metadata should be supported).

bkamins avatar Aug 13 '23 16:08 bkamins