lance
lance copied to clipboard
Temp manifest files are not cleaned up after commit using external manifest store
When using external manifest store, step 3 of commit does this copy:
COPY_OBJECT_STORE mydataset.lance/_versions/{version}.manifest-{uuid} mydataset.lance/_versions/{version}.manifest
https://lancedb.github.io/lance/format.html#external-manifest-store
This leaves the file mydataset.lance/_versions/{version}.manifest-{uuid}
which is unused and never cleaned up. We should remove this file as part of the commit procedure.
Also, when we run dataset cleanup
, these temporary files should be removed form object store.