lance icon indicating copy to clipboard operation
lance copied to clipboard

Temp manifest files are not cleaned up after commit using external manifest store

Open albertlockett opened this issue 11 months ago • 0 comments

When using external manifest store, step 3 of commit does this copy: COPY_OBJECT_STORE mydataset.lance/_versions/{version}.manifest-{uuid} mydataset.lance/_versions/{version}.manifest

https://lancedb.github.io/lance/format.html#external-manifest-store

This leaves the file mydataset.lance/_versions/{version}.manifest-{uuid} which is unused and never cleaned up. We should remove this file as part of the commit procedure.

Also, when we run dataset cleanup, these temporary files should be removed form object store.

albertlockett avatar Mar 12 '24 13:03 albertlockett