trino icon indicating copy to clipboard operation
trino copied to clipboard

Iceberg-connector: Delete the oldest tracked version metadata files after commit

Open 7hong opened this issue 1 year ago • 0 comments

Description

When using the iceberg table, the xxx-metadata.json file is generated each time commit is executed. In the iceberg table, we can automatically clean the previous metadata file through configuration.

write.metadata.delete-after-commit.enabled = true write.metadata.previous-versions-max = 10 However, the metadata file is not automatically cleaned in trino. A large number of metadata.json files are left on the hdfs.

Different from #20863 ,I have followed the configurations in iceberg (write.metadata.delete-after-commit.enabled and write.metadata.previous-version-max ). Instead of adding a new configuration, this keeps it compatible with iceberg.

Release notes

( x) Release notes are required, with the following suggested text:

## Iceberg connector

* Delete the oldest tracked version metadata files after commit. ({issue} #19582 ). 

7hong avatar Oct 17 '24 07:10 7hong