delta icon indicating copy to clipboard operation
delta copied to clipboard

[UNIFORM] Remove the `setSchema` patch

Open Fokko opened this issue 1 year ago • 1 comments

This patch aims to align the field-ID assignment between uniform and Iceberg. This way we can use the public Iceberg APIs to evolve the schema, instead of enforcing the schema through a private API.

The difference is that Iceberg does not have true pre-order traversal, but when it encounters a struct, it reserves an ID for each field in the struct before continuing the traversal.

Iceberg:

1: struct
  2: struct
    4: long
    5: timestamp
  3: int

Delta:

1: struct
  2: struct
    3: long
    4: timestamp
  5: int

This PR changes Delta's assignment behavior, so we don't need the patch anymore.

Which Delta project/connector is this regarding?

  • [x] Spark
  • [x] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [ ] Other (fill in here)

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Fokko avatar Aug 01 '24 12:08 Fokko

cc @amogh-jahagirdar

Fokko avatar Aug 05 '24 17:08 Fokko