[Bug] DeleteDocumentHandler does not delete all records from AI Search

[gabrielgirbacea]

Context / Scenario

Steps to reproduce:

1. Set-up Kernel Memory locally with Swagger
2. Configure it to use AI Search
3. Import a larger file (size >200 kb - only text document), something to generate more than 50 partitions in AI Search. Ex: lotr.txt
4. Call the "delete document" endpoint

Expected result: Deletes all the records/partitions from AI Search for a document id. Actual result: Some records/partitions are not deleted.

A few examples of how many records were deleted per different sizes of the file:

• for 83 partitions, 33 remain undeleted. After the second delete, 0.
• for 63 partitions, 13 remain undeleted. After the second delete, 0.
• for 332 partitions, 143 remain undeleted. After the second delete is triggered, 33 remain undeleted. After the third delete, 0.

Please let me know if I can provide more details. Thank you.

What happened?

Calling the "delete document" endpoint should delete all records from AI Search for the provided document id, regardless of how many records there are stored.

Note: When importing a file for an existing document id, SaveRecordsHandler is deleting all the existing records, as expected, before adding the new records.

Importance

edge case

Platform, Language, Versions

C# - .NET 8.0 Windows 11 Visual Studio Professional 2022 Kernel Memory (latest - 0.94.241201.1)

Relevant log output