Tair document store

[dongqqcom]

Is your feature request related to a problem? Please describe. I think it would be great to have a document store which makes use of Tair in a similar way to the Milvus document store already implemented. Tair is an enterprise-level in-memory database developed by Alibaba, providing out-of-the-box cloud-native database services. Tair supports online expansion and contraction, including horizontal and vertical expansion. In addition, high availability of active and standby systems is achieved, and the failover delay is short. Tairvector is an extended data structure self-developed by Tair, which provides real-time high-performance vector database services in pure memory mode. In terms of the HNSW real-time online method, the reading and writing time is shorter than that of Milvus. It supports combinations of various algorithms and data types, and supports Cosine, IP, L2 and Jaccard distance functions.

Describe the solution you'd like A seamless Tair document store which integrates with deepset, same functionalities than MilvusDocumentStore. Functions for documents and labels such as insertion, counting, filtering, TopK query and deletion will be implemented.

Describe alternatives you've considered An adhoc / separate implementation without deepset integration would be possible but a tighter integration would make things easier.

Additional context N/A

[anakin87]

Hey @dongqqcom...

Thank you for the idea and the implementation provided in #5358! :blue_heart:

We have several DocumentStores in Haystack, and it has become costly and complex to maintain them all within the core Haystack project. As such, we are encouraging contributors to create new DocumentStores in external projects, such as QdrantDocumentStore. Also MilvusDocumentStore has been moved to an external repository.

So, we invite you to create an external repository (and package), similar to the Qdrant example. Then we can add it to Haystack integrations and show it on the Haystack web site.

What do you think? Does this sound like a good plan?

[ghost]

Hey @dongqqcom...

Thank you for the idea and the implementation provided in #5358! 💙

We have several DocumentStores in Haystack, and it has become costly and complex to maintain them all within the core Haystack project. As such, we are encouraging contributors to create new DocumentStores in external projects, such as QdrantDocumentStore. Also MilvusDocumentStore has been moved to an external repository.

So, we invite you to create an external repository (and package), similar to the Qdrant example. Then we can add it to Haystack integrations and show it on the Haystack web site.

What do you think? Does this sound like a good plan?

Thank you for your reply. We will create an external repository like Qdrant soon.

[dongqqcom]

Hey @anakin87 ... Thank you for your reply. Can we just integrate tair into the repository haystack-extras like MilvusDocumentStore instead of creating another repository?

[anakin87]

It is better to create another repository, so you will be free to maintain it, release new versions... without having to depend on us for these operations. (The plan for the future is to also move MilvusDocumentStore to an external repo).

If you create a repository for the TairDocumentStore, we can help you set up the corresponding page in Haystack integrations...

[masci]

If this is still relevant, please open a feature request on https://github.com/deepset-ai/haystack-core-integrations/issues

[dongqqcom]

你好，我已收到你的邮件，会尽快给你回复。