iceberg-python icon indicating copy to clipboard operation
iceberg-python copied to clipboard

Published 20 hours ago • verified publisher icon apache

feat: refresh table when committing to support concurrent appends

Open YinZheng-Sun opened this issue 8 months ago • 3 comments

Rationale for this change

Support for Concurrent Append Operations

Refresh Table

  1. If changes in the table's metadata are detected (indicating that other transactions have been committed during this period), update the table's metadata and the current parent_snapshot_id.

  2. If no changes are detected, proceed with the normal commit.

Are these changes tested?

UT

Are there any user-facing changes?

YinZheng-Sun avatar Apr 07 '25 03:04 YinZheng-Sun

Thanks @YinZheng-Sun for working on this. It looks like it is a duplicate of https://github.com/apache/iceberg-python/pull/1772

Fokko avatar Apr 07 '25 04:04 Fokko

Thanks @YinZheng-Sun for working on this. It looks like it is a duplicate of #1772

Yes, this PR is inspired by #1772 and #1678. I want to accelerate the progress of this feature and have fixed the UTs

YinZheng-Sun avatar Apr 07 '25 05:04 YinZheng-Sun

I've created some subtasks on https://github.com/apache/iceberg-python/issues/819 that will help us implement the required validation functions that we can invoke to check that no conflicting commits have been made between two snapshots. @YinZheng-Sun would you be interested in helping out with some of those implementations?

sungwy avatar Apr 18 '25 01:04 sungwy