iceberg-python
iceberg-python copied to clipboard
apache
Add Support for Dynamic Overwrite
Added support for dynamic overwrite leveraging delete and fast-append(counterpart in Iceberg Spark).
Several follow-ups:
- to support current spec with transformed fields. Should be easy but due to the number of transforms, this takes some time. Will add them bit by bit in follow-up prs.
- could consider whether to raise userwarning when no delete is executed. Because from prespectives of users of dynamic overwrite, they should not worry about whether it is an pure append or a partition replacement.
@jqin61 Sorry for the slow review, I was doing some other stuff as well. Can you fix the merge conflicts? I think this looks good to go 👍
@jqin61 Sorry for the slow review, I was doing some other stuff as well. Can you fix the merge conflicts? I think this looks good to go 👍
Thank you Fokko! Sorry for the delay, I was extremely busy recently, I will get some time next weekend to fix the comments, add tests and fix the documentation. I will also move the transform support out of the scope of this pr due to its complexity, will send you details about it soon.
Thank you for making this contribution @jqin61 ! I'll leave this PR open for another review, especially given that it introduces a new table commit API
Hi @Fokko - this PR looks good from my end.
Would you have some time to take a look? Since this is a new API (which comes with another level of caution), I'd love to get your review before we merge in @jqin61 's awesome work
@jqin61 @sungwy Sorry for leaving this hanging, I'll do a review first thing tomorrow 👍
@jqin61 Do you have time to follow up on the last few comments? Would be great to get this in 👍
@Fokko @sungwy Thank you for the review and the suggestions! I fixed the latest comments and let's rerun CI and merge it if looks good to you.