CosmosDB icon indicating copy to clipboard operation
CosmosDB copied to clipboard

Add a function to deploy and execute bulkDelete stored proc for a query

Open jasonchester opened this issue 6 years ago • 2 comments

Bulk Delete Feature

Bulk deleting documents in cosmos db is not currently supported and the feature for bulk deleting all documents within a partition is not planned.

  • https://stackoverflow.com/questions/52708076/how-to-delete-all-data-in-a-partition
  • https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/17296813-add-the-ability-to-delete-all-data-in-a-partition

Microsoft provides an example stored procedure for deleting documents in bulk.

  • https://github.com/Azure/azure-cosmosdb-js-server/blob/master/samples/stored-procedures/bulkDelete.js

A function that would simplify the multiple steps needed to deploy bulk-stored procedure and execute it in a loop should be added which performs the following steps.

proposed syntax

Remove-CosmosDbBulkDocuments -Context '...' -Database 'db' -CollectionId 'events'  -PartitionKey 'users' -query 'SELECT * FROM c WHERE c.Date < "1/1/2019" '

execution steps

  1. Deploy the stored procedure to the Collection
  2. Calculate total documents that will be deleted and starting time of the operation
  3. Execute stored procedure in a loop until there are no more documents to delete
do {
    $result = Invoke-CosmosDbStoredProcedure -Context $ctx -CollectionId 'test' -PartitionKey 'partition-1' -Id 'bulkDelete' -StoredProcedureParameter @( 'select * from c') 
} while($result.continuation)

each iteration should display the following statistics

  • number of deleted documents
  • documents remaining / documents total
  • elapsed time
  • remaining time
  1. optionally the stored procedure might get removed when completed

jasonchester avatar May 04 '19 03:05 jasonchester

Good idea @jasonchester.

Suggest we could extend the existing Remove-CosmosDbDocument function with a -Query parameter instead of implementing a new function. Within the Remove-CosmosDbDocument function it might call out to a private bulk delete function if required.

Alternately the name `Remove-CosmosDbDocumentBulk' could be used. That would then cause the Cmdlet autocomplete to show the cmdlet within the CosmosDbDocument namespace. Also PS guidelines recommend not using plural names (e.g. no 's' on the end).

PlagueHO avatar May 04 '19 03:05 PlagueHO

looks like Microsoft has begun work to enable removing all documents in a partition.

https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/17296813-add-the-ability-to-delete-all-data-in-a-partition

jasonchester avatar Mar 16 '20 18:03 jasonchester