deephaven-core
deephaven-core copied to clipboard
iceberg primitive array support
We are currently unable to deserialize byte[], and likely other primitive array types.
Value: io.deephaven.engine.table.impl.locations.TableDataException: Unsupported iceberg column type LIST
at io.deephaven.iceberg.util.IcebergCatalogAdapter.convertPrimitiveType(IcebergCatalogAdapter.java:161)
at io.deephaven.iceberg.util.IcebergCatalogAdapter.fromSchema(IcebergCatalogAdapter.java:108)
at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTableInternal(IcebergCatalogAdapter.java:506)
at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTable(IcebergCatalogAdapter.java:380)
at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTable(IcebergCatalogAdapter.java:394)
While I don't think we necessarily need nested array support, we should probably support at least single primitive arrays?
We might consider that the catalog type (as written by pyiceberg) for a byte[] type would be better represented as a BinaryType (as opposed to List
That said, we should consider the same scenario but with short[].
Moved to https://deephaven.atlassian.net/browse/DH-18260