.Net: Weaviate memory connector http call optimization

[atiq-bs23]

Motivation and Context

To improve efficiency and performance by reducing the number of HTTP requests during batch deletions in RemoveBatchAsync.

Description

The updated RemoveBatchAsync method in WeaviateMemoryStore now sends a single HTTP request to delete multiple keys at once, replacing the previous approach of sending individual requests for each key. This change minimizes network overhead and enhances performance.

Doc: Batch delete objects

Contribution Checklist

• [ ] The code builds clean without any errors or warnings
• [ ] The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• [ ] All unit tests pass, and I have added new tests where possible
• [ ] I didn't break anyone :smile:

[atiq-bs23]

@westey-m, @RogerBarreto Can you please help me to test this current change?

[atiq-bs23]

@markwallace-microsoft all hard-coded null checks replaced.

[westey-m]

@westey-m, @RogerBarreto Can you please help me to test this current change?

@atiq-bs23, looks like there is an issue with the batch request model. I'm getting 422 (Unprocessable Entity) when running the integration tests.

To run them yourself, check out https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/IntegrationTests/Connectors/Weaviate/WeaviateMemoryStoreTests.cs

Steps are:

1. Start up the weaviate docker image, using the instructions here or the docker compose yml that's in the same folder as the tests.
2. Set the skipReason to null on line 23 of the test file, so that it doesn't skip the tests.

Let me know if you get stuck.

[atiq-bs23]

@westey-m based on their doc Batch delete objects , it seems the body is okay to me.

Can you please check if there is anything I am missing?

To generate the body please take the attachment Program.txt ->MAKE IT Program.cs -> Run as a console app Program.txt

JSON BODY { "dryRun": false, "match": { "class": "myCollection", "where": { "operator": "ContainsAny", "path": [ "sk_id" ], "valueStringArray": [ "key1", "key2", "key3" ] } } }

In the meantime I will check weaviate docker image.

[atiq-bs23]

@westey-m Integration Test Passed I have got the issue and added a fix. Instead of passing the Weaviate-friendly class name, I was mistakenly passing the user input directly.

dotnet/src/Connectors/Connectors.Memory.Weaviate/WeaviateMemoryStore.cs#L394

Apologies for the late reply. We experienced an internet outage across Bangladesh for the past few days.

[atiq-bs23]

@RogerBarreto could you please review this implementation when you have a chance? unit test and integration test passed.

[markwallace-microsoft]

@atiq-bs23 we have recently announced a new Microsoft.Extensions.VectorData.Abstractions package which we have developed in partnership with the .NET team. We have also started migrating our memory connectors to use these new abstractions and plan to obsolete the old implementations. So we are not taking any new changes to the older memory connectors. Thanks for your interest in the Semantic Kernel and your contributions.