cassandra-medusa
cassandra-medusa copied to clipboard
thelastpickle
Slow deletion of orphan files
After running a purge/delete of backups, medusa tries to remove SSTable files that are not referenced by any existing backups (source). This process can be lengthy with a high amount of files to be deleted, not to mention the high mem usage as reported in #393. I'm running medusa with AWS S3 as the storage. I found that files are deleted one by one and I measured an upper bound of 20 deletions per second. This is slow given that in my case, medusa has to cleanup 500 000 files per fqdn, which gives me a total of 13 500 000 files to delete. It would take almost 8 days for everything to be deleted. I know that libcloud is quite limiting and doesn't seem to support batch deletes (AWS allows you to delete up to 1000 objects in one request) and it also seem that you have to first make a request to get the object before being able to delete it, even if you already know the object path.
My workaround has been to run medusa --fqdn $fqdn purge for all fqdn and in parallel.
┆Issue is synchronized with this Jira Story by Unito
