spark-utils icon indicating copy to clipboard operation
spark-utils copied to clipboard

Add table maintenance for iceberg table format

Open cccs-jc opened this issue 4 years ago • 0 comments

spark-utils has table maintenance macros for DeltaLake tables.

For example these macros list the tables in a schema and perform vacuum. https://github.com/dbt-labs/spark-utils/blob/b41aab4e63acb205d11bf80e2a7bbe0dd63a3c6f/macros/maintenance_operation.sql

In the same manner, it would be great if spark-utils also had support for Iceberg table maintenance.

https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management Examples of maintenance could be

  • expire_snapshots
  • remove_orphan_files
  • rewrite_data_files
  • rewrite_manifests

cccs-jc avatar Apr 11 '22 14:04 cccs-jc