delta-rs icon indicating copy to clipboard operation
delta-rs copied to clipboard

fix: cast support fields nested in lists and maps

Open HawaiianSpork opened this issue 1 year ago • 1 comments

Description

The current implementation of cast only works for structs nested in structs. This PR adds supports for structs contained in other types (lists and maps). This PR also prevents cast from adding nullable column if the field is not nullable, instead it will throw an error.

Note: This is only a partial solution which would let you merge schema with nested missing columns, it does not allow delta-rs to read the merged schema (though Spark can). To read the merged schema will require another change where delta-rs defines its own datafusion parquet schemaAdapter.

HawaiianSpork avatar May 27 '24 02:05 HawaiianSpork

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

github-actions[bot] avatar May 27 '24 02:05 github-actions[bot]

Please add also a bunch of python tests

ion-elgreco avatar May 29 '24 15:05 ion-elgreco

Please add also a bunch of python tests

@ion-elgreco I'm confused as to what a Python set of tests would do that the tests in Rust don't already do? :confused:

rtyler avatar May 29 '24 19:05 rtyler

Please add also a bunch of python tests

@ion-elgreco I'm confused as to what a Python set of tests would do that the tests in Rust don't already do? :confused:

We mostly need to check the reader side with pyarrow datasets

ion-elgreco avatar May 29 '24 19:05 ion-elgreco

@HawaiianSpork thanks for the PR! can you rebase please so we can merge

ion-elgreco avatar Jun 04 '24 10:06 ion-elgreco

@ion-elgreco thank you. Sorry, I did not get around to writing python tests. The code has been rebased.

HawaiianSpork avatar Jun 05 '24 04:06 HawaiianSpork