deephaven-core icon indicating copy to clipboard operation
deephaven-core copied to clipboard

Iceberg column rename handling

Open devinrsmith opened this issue 1 year ago • 2 comments

Deephaven does not currently handle Iceberg column renames.

Here is an example to produce a set of data that illustrates the issue https://gist.github.com/devinrsmith/a6d537e5dce3ff0b25ba4b9c62d80ada which will first create the table, add some data, rename a column, and add some more data.

When read into Deephaven, only the latest data files have their data represented:

Screenshot from 2024-09-24 10-02-30

devinrsmith avatar Sep 24 '24 19:09 devinrsmith

https://github.com/apache/iceberg/blob/main/format/spec.md#schema-evolution should be consulted in general for issues about how schema evolution and tracking is supposed to happen.

https://www.mail-archive.com/[email protected]/msg02236.html also talks a bit about Iceberg field-id and Parquet field_id.

devinrsmith avatar Sep 25 '24 15:09 devinrsmith

Potentially related to #6124

devinrsmith avatar Sep 30 '24 21:09 devinrsmith

Moved to https://deephaven.atlassian.net/browse/DH-19178

malhotrashivam avatar Apr 09 '25 14:04 malhotrashivam