s3backer
s3backer copied to clipboard
archiecobbs
Multiple Object Delete Support
When deleting multiple blocks the network usage is suboptimal since each block is deleted with a single delete request. It would be more effient to make a single delete request with multiple objects (reference http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html).
The --erase flag is the most impacted scenario, but normal usage (where filesystem is mounted with discard) could also get a boost (eg rm -rf /this/folder, or when deleting big files).
Another quirk seems to be the "max-keys" parameter of the GET /?marker.... http request (that is actually fixed at 256). It would be better to make it an option in the command line, and set its default to the system maximum (1000). This gives network efficiency during the initial block listing.
Hi @xmas79, I'm reopening this issue because it is valid and a good improvement. I have been too busy to look at it but will do so eventually. Thanks.
Hi Archie, sorry for closing this issue. I thought you could simply see the patches in the fork and merge them when reviewed the code and tested the functionalities I already implemented.
And sorry for being so confusing in senting patches, but when I tried to implement the bulk delete I had to speed up the development process: testing a bulk delete means create (and put data!) and then delete a bunch of (big) filesystems, so I implemented other patches as well even before the bulk delete was functional.
Take you time to review all the changes, and thanks.
The situation has improved since this bug was filed.
In 016c3fe6 the --erase function deletes in parallel, and we list blocks 1000 at a time.
s3backer still only deletes one block per HTTP request however.