iceberg
iceberg copied to clipboard
[WIP] Handle table evolution using schemaId mapping
Scenario:
CREATE TABLE %s (id bigint, data string, category string) USING iceberg PARTITIONED BY (category) TBLPROPERTIES('format-version' = '2')
ALTER TABLE %s DROP PARTITION FIELD category
ALTER TABLE %s DROP COLUMN category
-- throws NPE as mentioned in #5399
Fix: Fields in a partition spec can belong to an older schema. So, keep a schema id mapping in each field of partitionSpec. PartitionSpec should contain all the schema lists from TableMetadata.
Testcase:
Added a test case to cover this scenario in TestAlterTablePartitionFields.java
- TODO: test and handle backward compatibility
- TODO: Same problem can exist for sort orders also. Need to modify spec there as well.
Fixes #5399
@rdblue, @RussellSpitzer : This PR needs some more work (mentioned TODO in the description). But I would like to have early feedback about this Iceberg spec modification.
The actual issue scenario is passing with this change.