amoro
amoro copied to clipboard
[Bug]: keyed mixed iceberg format table always failed to expire snapshots of it's basetable
What happened?
In a MixedTable (composed of BaseTable and ChangeTable), both tables share the same table metadata. When performing snapshot expiration:
- The change table's snapshot cleanup completes and commits, which updates the shared table metadata (incrementing the meta_version).
- The base table's snapshot cleanup then attempts to commit, but it still holds a stale reference to the table metadata with an outdated meta_version.
- This causes the base table's commit to fail with a meta_version conflict error, since the metadata version has already been incremented by the change table's commit.
Root Cause: The BaseTableMaintainer and ChangeTableMaintainer both operate on the shared table metadata, but when changeMaintainer.expireSnapshots() commits, it updates the metadata version. The baseMaintainer instance still holds the old metadata version, leading to a conflict when it tries to commit its snapshot expiration.
Expected Behavior: After the change table's snapshot cleanup commits, the base table should refresh its metadata reference to get the latest version before attempting its own snapshot cleanup commit.
Affects Versions
master
What table formats are you seeing the problem on?
Mixed-Iceberg
What engines are you seeing the problem on?
AMS
How to reproduce
- create mixed iceberg table and writon with flink cdc
- enable ams auto expire snapshots with default setting (ttl can be shorter for reproduct)
- after expiring process finished, change table success remove snapshots and base table keeps all the snapshots
Relevant log output
Anything else
No response
Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct