Support IsolationLevels and Concurrency Safety Validation Checks

[sungwy]

Feature Request / Improvement

Support enforcing Isolation Levels from specified snapshot ID

https://iceberg.apache.org/docs/latest/spark-configuration/#write-options

There's been a lot of continued interest in using multiple PyIceberg applications concurrently and having proper support for optimistic concurrency.

I think the best place to start is through the implementation of the individual validation functions

Once this is complete, we'll be able to introduce the Isolation Levels and correctly implement the validation logic in the _OverwriteFiles snapshot producer, similarly to the Java implementation

[jqin61]

Hi I am interested in working on it!

[sungwy]

Some relevant links to the Java implementation

• validateNewDataFiles flag -> MergingSnapshotProducer: validateAddedDataFiles
• validateNewDeletes flag -> MergingSnapshotProducer: validateDeletedDataFiles

[guptaakashdeep]

Hey @sungwy I would like to contribute by working on these.

Is there any of these that I can pick and starts looking into it like any of the initial validation implementation ?

[sungwy]

@guptaakashdeep yes, I don't think there's a particular order we should implement these with, so please feel free to assign yourself to the one you find most interesting!

Sung

[guptaakashdeep]

Thanks @sungwy ! Do we have any already existing class where I can implement these Validation functions or should we just add directly in snapshot.py ?

[sungwy]

I think we could create a new module as pyiceberg.table.update.validate.py and add these validation checks there. What do you think @guptaakashdeep ?

[guptaakashdeep]

Sounds good @sungwy !!

[jayceslesar]

@guptaakashdeep @sungwy see https://github.com/apache/iceberg-python/pull/1935 which should be the building blocks needed to crank out the 4 Sub-issues

[jayceslesar]

Also going to crank out a manifest group implementation today

Edit: @sungwy it looks like the manifestgroup.entries method is extremely similar to the DataScan defined in Table __init__.py file...What do you think?

[cnatsis]

Is there any update on this feature?

[dyami0123]

Also curious if there is any movement on this PR. I currently have some workarounds for concurrent writes implemented but they are very inefficient.