mongo-arrow icon indicating copy to clipboard operation
mongo-arrow copied to clipboard

Support for nested ObjectIDs in polars conversion

Open sibbiii opened this issue 8 months ago • 3 comments

Hi,

_arrow_to_polars currently has no support to cast extension types for nested fields. This prohibits ObjectIDs to be read in case they are in nested fields.

I could not manage the conversion with the original code, but I found a way to using arrow_table_without_extensions = arrow_table.cast(schema_without_extensions) to cast the schema of the whole table in one go.

The schema_without_extensions is created recursively from the old schema. Support for lists is still to be added, should not be that hard, maybe I try tomorrow.

I am not an expert in apache arrow. My world is Pandas and Polars. I have wrote some unit tests locally to test the code, but I do not feel confident that I have not overlooked something, so please review carefully.

#219

sibbiii avatar Jun 17 '24 23:06 sibbiii