apache
Support Snapshot Expiration Operation
Feature Request / Improvement
Support Maintenance operations on PyIceberg: https://iceberg.apache.org/docs/1.4.0/maintenance
All operations except for data file compaction are metadata-only or file system operations, so supporting them on PyIceberg may be a small lift
Metadata rewrites are already discussed in https://github.com/apache/iceberg-python/issues/270
I think snapshot expiration is one that we should also focus on. I think https://github.com/apache/iceberg-python/issues/511 is a prerequisite to ensure we have a structured way of listing all the metadata so we know which files we can remove. This would then just be operations on a PyArrow Table 🎉
@Fokko
just to make sure, currently there is not support in pyiceberg for snapshot expiration?
If so, do you have any suggestions of how to remove old data (I'm writing data using pyiceberg to AWS Glue)
I started a discussion on the mailing list about the delete orphan files and meanwhile I'm studying the expire snapshots.
Hello everyone, I need some help with this.
During the implementation process, I noticed that we lack some features that exist in the Java implementation. The first one I want to discuss is the TableOperations class. In the RemoveSnapshots class, the io, current, refresh, and commit methods are used.
Currently, I can use the io from our Table implementation, and I can get the same behavior through Table to get and refresh the TableMetadata. However, I haven't found an implementation to commit TableMetadata.
The problem is that we require an implementation for each catalog. Currently, the Java implementation uses the TableOperations interface for the REST catalog, but the other catalogs (Snowflake / JDBC / Glue) use a different interface with the same methods.
My question is: how should we handle the commit process? I don't have a solid opinion on how we can solve this.
Just to clarify, this commit method is only for updating the table metadata.
cc/ @Fokko @HonahX @kevinjqliu @sungwy
After reading my question again, I'm not sure if I was clear enough. I want to discuss the design of the implementation in more detail.
I'm currently creating a class (ExpireSnapshot) to handle the expiration process (similar to RemoveSnapshots in Java) and also classes that will manage file deletion based on the updated metadata (ReachableFileCleanup and IncrementalFileCleanup). I believe this design makes sense, but my question is about where the TableMetadata commit should live. Does it make more sense to create a TableOperations class (with an implementation for each catalog), or should we add a method to each catalog that handles the commit process?
how should we handle the commit process?
The current way to commit involves the public commit_table method implemented for each catalog (ie SQL catalog). It takes in requirements and updates, similar to the RESTTableOperations::commit function.
I want to discuss the design of the implementation in more detail.
Do you mind starting a doc on what you've found so far? It would be helpful to figure out what is needed for snapshot expiration before diving into how to implement it
@kevinjqliu Thank you, I will start writing a document next week detailing what I found and explaining the differences in the commit operation and what we don't have in Python.
@kevinjqliu I believe that I now understand the differences in how we perform TableMetadata updates in Python versus how it's done in Java. I think that the set of classes describing the changes, along with the use of single dispatch, makes it a bit harder to understand compared to the Java builder strategy. I will continue working on the document to discuss other points.