Add new options to delete-bucket command

[metadaddy]

Users sometimes need to delete a bucket and all the objects in that bucket. It is laborious to write a script to do so, and the use of b2 sync is not obvious. Add the following options to delete-bucket:

• --recursive delete all file versions in the bucket, then delete the bucket itself. The user is prompted for confirmation, unless the --force option is supplied. The command prints the details of each file version (using the same format as the ls command) as it is deleted, unless the --quiet option is specified.
• --force do not prompt for confirmation when doing a recursive delete.
• --quiet do not print details of file versions as they are deleted.
• --deleteKeys delete application keys scoped to this bucket, since they will be redundant.
• --deleteReplications delete replication rules associated with this bucket.
• --deleteAll abbreviation for --recursive --deleteKeys --deleteReplications

The following description is in terms of B2 Native API operations, though the command will call the equivalent methods in b2-sdk-python:

When recursive is specified, after the confirmation step, the command will call b2_list_file_versions on the bucket once, and loop through the results, calling b2_delete_file_version for each file version and, if -quiet was not specified, printing the details of the file version. If either b2_list_file_versions or b2_delete_file_version returns an error, the command will terminate, reporting that error.

Once the command has called b2_delete_file_version for each file version returned by b2_list_file_versions, it will call b2_delete_bucket to delete the bucket itself, reporting any error returned by that operation.

Note that b2_delete_bucket may fail due to the bucket not being empty, as files may be uploaded to the bucket after b2_list_file_versions was called. In this case, the command will call b2_list_file_versions again, accompanying the error message from b2_delete_bucket with the message One or more files were uploaded while the bucket contents were being deleted: and the results of the b2_list_file_versions call.

[metadaddy]

@ppolewicz Please assign this one to me. I'll write some code and submit a PR. Thanks!

[ppolewicz]

This duplicates in the large portion with the cli counterpart of https://github.com/Backblaze/b2-sdk-python/pull/361 (already merged but not yet released due to a couple more PRs landing on the sdk side next week).

The basic idea would be to run b2 rm bucket-name '*' && b2 delete-bucket bucket-name and make it the responsibility of the user to make sure they are not uploading any more stuff to the bucket while they are trying to remove it completely. Putting both in the same command could make sense, especially if we added --attempts 10 to try it many times in case some uploads keep landing - that'd be hard to achieve in a different way (without a special command to support it).

As for internals, it's a bit more complicated than in your design - the wildcard removal system uses a thread pool in order to increase performance - removing a file version is an io-bound call and putting those in python threads can make things much faster (here I believe it's nearly 10 times faster with default settings). Furthermore the streaming and backpressure are important considerations - a b2 bucket can contain a bazzilion objects and just their version ids might not fit within memory of the system that is trying to clean it. Anyway - that's merged now. I suggest that you wait for the cli counterpart of the mentioned PR to land and then chain that functionality in an optional mode of "delete-bucket", add a couple of tests and have a good feature.

I'm sorry about the review queue buildup, I've been horribly sick and could not review long PRs while two people were coding. I hope that clears up a lot in the next few days.

[metadaddy]

I suggest that you wait for the cli counterpart of the mentioned PR to land and then chain that functionality in an optional mode of "delete-bucket", add a couple of tests and have a good feature.

Sounds good to me!

I've been horribly sick and could not review long PRs while two people were coding. I hope that clears up a lot in the next few days.

Take it easy, and get well soon!

[metadaddy]

Closing now that 3.7.0 is available with rm command.

[metadaddy]

@ppolewicz Reopened and added new options.