iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

Support validate and related logic in Snapshot

Open CTTY opened this issue 7 months ago • 3 comments

Is your feature request related to a problem or challenge?

To detect conflicts, we need to have validate and related logic to scan manifests and validate against history.

References:

  • java: https://github.com/apache/iceberg/blob/c2478968e65368c61799d8ca4b89506a61ca3e7c/core/src/main/java/org/apache/iceberg/BaseRowDelta.java#L128
  • python: https://github.com/apache/iceberg-python/pull/1935

Describe the solution you'd like

Add related logic to SnapshotProduceOperation so different actions like RowDelta, OverwriteFiles, and RewriteFiles can use and override (if needed)

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

CTTY avatar May 17 '25 05:05 CTTY

predicate/residual evaluator and manifest filter are also important (to reduce conflict rate) but they could be completed separately, wdyt? @Xuanwo @liurenjie1024 @jonathanc-n

CTTY avatar May 17 '25 06:05 CTTY

Yes those can be completed separately, I will try to get residual evaluator done (a bit difficult since the schema + filters in rust are a bit different)

jonathanc-n avatar May 17 '25 22:05 jonathanc-n

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

github-actions[bot] avatar Nov 14 '25 00:11 github-actions[bot]