infiniflow
[Question]: Multilingual support between embedding knowledge base, retrieval testing, search, and assistant chat
Describe your problem
As this project has a Chinese/English focus I tried to experiment with a bilingual test case.
So the source document is in Chinese:
Embedding is done with maidalun1020/bce-embedding-base_v1, which I understood to be a Bilingual and Crosslingual Embedding model.
I work under the assumption that it means while the source document is in Chinese, I will be able to perform retrieval testing, search, and chat in English should the semantic meaning of a chunk matches. Obviously the LLM deployed (Gemini) needs to be bilingual as well which is the case.
However that is not what I have experienced with.
Retrieval testing: Always return with "no data"
Search: No result
Chat: Knowledge base is empty
Please advise if multilingual support is available in Ragflow, or if what has attempted wasn't the correct approach for such a purpose? Thanks.
I would second this question. I tried multi-language use case (one knowledge base, documents in two languages, embedder e5-medium that is multi-lingual). When I asked question in English- only English documents are used for reference, when I am asking in second language - it uses only second language documents.
Upon further experiments I found that the limitation is more to do with the RAG process rather than the LLM model. Basically an embedded vector from English questions will not retrieve any embedded vector with Chinese data, thus leaving any subsequent LLM interaction irrelevant. However according to the description of BCEmbedding model:
EmbeddingModel handle bilingual and crosslingual retrieval task in English and Chinese
So why would this become a hurdle when adopted and utilized within RAGFLOW?
Just thinking aloud: Did you test the process outside of rag flow? May be issue is in embeddings model , and not on rag flow side ? I will also test some proprietary embedders available via api, will come back if something interesting comes up
Multilingual search is not supported well so far.
Hi, what if I integrate specific embedding model for my target langauge? I know via GUI this is possible, also I can deploy that model on the same host and it should work but also I see the parameter "langauge" in the "knowldge base" configuration, does it mean that one of these 3 langauges will be used as an input parameter for embedding model when we process the chunk. If it so, then how I can add the 4th langauge into the list?
Also the model like "all-MiniLM-L6-v2" supports different langauges and maybe if we can specify the original text langauge as an input parameter for embedding process then it could be an easy win for a timebeeinng. This is just a question in case if such workaround is possible, and if yes could you please help to find our where in the code it needs to be changed ...
thank you
In order to verify this problem, I use the cohere.embed-multilingual-v3 vectorization model to perform a vectorization comparison test using a python script locally. The similarity between Chinese and English is 0.8 and 0.76, but it cannot be retrieved in ragflow using English
Thank you so much for your suggestion! 🙏 We really appreciate your input.
In our latest version, we've added Multilingual support! 🌍✨ Please give it a try and let us know if it helps resolve your issue. We hope this feature provides a better experience for you!
Once again, thank you for your continued support of RAGFlow! We’re always excited to hear your thoughts and ideas, and we look forward to making the product even better with your help. 😄 @predoctech
