polarsignals
Support complex Parquet types
Currently, ArcticDB only supports simple columns, no repeated or struct columns, it would be nice if it would also support more complex types.
This would require two things:
- Allow merging of schemas with complex types (with dynamic columns).
- Extend conversion of complex (by complex I mean arbitrarily deeply nested struct and repeated types) parquet types to arrow types.
1 should be slightly easier now since parquet-go now implements conversions, which happens to perfectly suit the dynamic columns use-case: https://github.com/segmentio/parquet-go#evolving-parquet-schemas-parquetconvert. 1 can also be implemented before 2 without breaking anything existing.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.