iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard
verified publisher icon apache

Partition Writer Support Part 1: add partition splitter

Open ZENOTME opened this issue 9 months ago • 2 comments

Which issue does this PR close?

This PR is part 1 to close #342.

What changes are included in this PR?

The partition writer support will be separate into three PR:

  1. add partition splitter which used to split the record batch based on partition value
  2. add fanout partition writer which will compute the partition value of input and split them
  3. add precompute partition writer which will use the partition value provided by user

This PR is the first part.

Are these changes tested?

ZENOTME avatar Mar 05 '25 07:03 ZENOTME

This PR may conflict with #1014. But I'm not sure whether #1014 can be reviewed and merge recently. I'm ok to merge any one first and I will fix the conflict later. cc @liurenjie1024 @jonathanc-n @Fokko @Xuanwo @sdd

ZENOTME avatar Mar 05 '25 07:03 ZENOTME

@ZENOTME I should be able to take a look at this tomorrow

jonathanc-n avatar Mar 17 '25 04:03 jonathanc-n

This PR is ready to review. cc @liurenjie1024 @Fokko @Xuanwo @sdd

ZENOTME avatar May 01 '25 12:05 ZENOTME

Hi @jonathanc-n & @ZENOTME Could we please enable partition support in Iceberg Rust? Can we please expedite this if possible?

ranjanankur314 avatar May 12 '25 09:05 ranjanankur314

@ranjanankur314 I will try my best to give it another review today. At the end of the day we will need a committer's review

jonathanc-n avatar May 12 '25 14:05 jonathanc-n

Hi, I think this PR is ready to review. cc @liurenjie1024 @Xuanwo @Fokko @kevinjqliu @sdd

ZENOTME avatar Aug 25 '25 16:08 ZENOTME