semantic-kernel
semantic-kernel copied to clipboard
microsoft
Rewriting CosmosDB memory connector and memroy connector unit tests
Hi guys, I've improved the CosmosDB memory connector, before creating the PR I'm re-wrting the unit tests for memory connectors.
Improves for CosmosDB connector:
- Use serverside UDF for cosin simularity calculation, greatly reduce the network bandwith
- improve parition key implementation
- improve DoesCollectionExistAsync() and GetCollectionsAsync().
Improves to unit tests
- Create a base class targets to test IMemoryStore
- Imherit CosmosDBTester from the base class
This ensures every IMemoryStore implementation can has a consistent test cases and make unit test much more easier. The question here is I don't now if my work is duplicated with your guys? If you're already working on this please let me know. I will change my effort to other area.
Thanks
Use serverside UDF for cosin simularity calculation, greatly reduce the network bandwith
This is interesting and would be a major improvement over what we have today (which was mostly intended as a stopgap)
@awharrison-28 would you mind commenting on this? If it's not being actively worked on could we take @JohnMasen up on the offer of help here?
@JohnMasen there is currently no duplication of your work on our end. This is so appreciated, thank you for your efforts!
That's great, just made all unit tests passed. The basic features are all complete. Code optimization and cleanup in progress. Will create a PR after that. The latest change can be found at https://github.com/JohnMasen/semantic-kernel
@JohnMasen we can't wait to get your PR in. Let us know if you have any questions.
@evchaki the PR 536 was stucked, seems the integration test keep failing due to cosmosDB emulator was not installed on the test host. I've asked help in the PR but seems not caught any attention. you can find the lastest CosmosDB code at https://github.com/JohnMasen/semantic-kernel
@JohnMasen I apologize for the delayed response here. We will be deprecating CosmosMemoryStore very soon due to it's expense and inefficiencies. We would really like to have your IMemoryStore unit test abstraction though. Feel free to keep your own implementation of CosmosMemoryStore though.
@awharrison-28 Thanks for your response. you can get my latest code from my reporsitory https://github.com/JohnMasen/semantic-kernel . Due to lack of experience in Github PR, I can't solve the integratinon test soon. Please feel free to get & merge it to your main branch with your own account, all I want is solve the cosmosDB efficient issue.
By the way, I'm thinking a new 2 layer memory system, it separates the search and storage into 2 different parts. this allows us to save the memory data to DBMS and still have fast in-memory embedding search feature.
