iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard

[WIP] Handle table evolution using schemaId mapping

Open ajantha-bhat opened this issue 2 years ago • 1 comments

Scenario: CREATE TABLE %s (id bigint, data string, category string) USING iceberg PARTITIONED BY (category) TBLPROPERTIES('format-version' = '2') ALTER TABLE %s DROP PARTITION FIELD category ALTER TABLE %s DROP COLUMN category -- throws NPE as mentioned in #5399

Fix: Fields in a partition spec can belong to an older schema. So, keep a schema id mapping in each field of partitionSpec. PartitionSpec should contain all the schema lists from TableMetadata.

Testcase: Added a test case to cover this scenario in TestAlterTablePartitionFields.java

  • TODO: test and handle backward compatibility
  • TODO: Same problem can exist for sort orders also. Need to modify spec there as well.

Fixes #5399

ajantha-bhat avatar Aug 02 '22 12:08 ajantha-bhat

@rdblue, @RussellSpitzer : This PR needs some more work (mentioned TODO in the description). But I would like to have early feedback about this Iceberg spec modification.

The actual issue scenario is passing with this change.

ajantha-bhat avatar Aug 02 '22 17:08 ajantha-bhat