deephaven-core icon indicating copy to clipboard operation
deephaven-core copied to clipboard

iceberg primitive array support

Open devinrsmith opened this issue 1 year ago • 1 comments

We are currently unable to deserialize byte[], and likely other primitive array types.

Value: io.deephaven.engine.table.impl.locations.TableDataException: Unsupported iceberg column type LIST
	at io.deephaven.iceberg.util.IcebergCatalogAdapter.convertPrimitiveType(IcebergCatalogAdapter.java:161)
	at io.deephaven.iceberg.util.IcebergCatalogAdapter.fromSchema(IcebergCatalogAdapter.java:108)
	at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTableInternal(IcebergCatalogAdapter.java:506)
	at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTable(IcebergCatalogAdapter.java:380)
	at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTable(IcebergCatalogAdapter.java:394)

While I don't think we necessarily need nested array support, we should probably support at least single primitive arrays?

devinrsmith avatar Jul 30 '24 20:07 devinrsmith

We might consider that the catalog type (as written by pyiceberg) for a byte[] type would be better represented as a BinaryType (as opposed to List) in catalog.

That said, we should consider the same scenario but with short[].

devinrsmith avatar Jul 30 '24 20:07 devinrsmith

Moved to https://deephaven.atlassian.net/browse/DH-18260

malhotrashivam avatar Dec 19 '24 19:12 malhotrashivam