iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

feat: refine record batch project

Open ZENOTME opened this issue 9 months ago • 2 comments

This PR:

  • use SchemaWithPartner visitor to refactor RecordBatchProjector
  • use RecordBatchProjector in RecordBatchTransform
  • complete #405

Context

For read or write in iceberg-rust, we interoperate with arrow RecordBatch. There are similar project tasks when reading and write to project RecordBatch:

  • In equality delete, we need to project record batch according to field id
  • in read, the user may scan part of the column and we also need to project record batch according to field id
  • for projects in reading, the iceberg also defines project rules to fulfill the value when a column is missing. Like init-default

For now, these tasks are implemented in different structs, RecordBatchTransform, and RecordBatchProjector and there are redundant codes. After we introduce the SchemaWithPartner visitor in #731, we can unify the implementation under the "visitor" framework.

The project task is generally separated into two parts:

  1. accessor: retrieve the array of partners. We can custom accessor to achieve init-default. E.g. when fail to retrieve the array, accessor can construct the array according to init-default. And we also can custom accessor to achieve name-map.
  2. visitor: do some convert task. We can customize visitors to do the type of promotion here.

The PR try this work to refactor our RecordBatchTransform, and RecordBatchProjector. And it also complete https://github.com/apache/iceberg-rust/issues/405#issue-2354723597 using ArrowProjectionVisitor as @liurenjie1024 mention

ZENOTME avatar Feb 27 '25 09:02 ZENOTME

cc @liurenjie1024 @sdd @Xuanwo @Fokko

ZENOTME avatar Feb 28 '25 07:02 ZENOTME

Potential improvement: For now, we will trave the whole record in visitor pattern, but in some case we don't need to travese child column. E.g. the map column is passthrough, we don't need type promotion for its child so we hope to return map column directly and stop traversing its child.

ZENOTME avatar Feb 28 '25 07:02 ZENOTME