FlowiseAI
Added meilisearch retriever component
for the past 2 weeks , we needed to implement a retriever that utilized meilisearch for hybrid search powered retrieval , the component was created as a custom retriever, you either search an existing index using meilisearch and retrieve context for your LLM or you can upsert an existing or a new index to include new files that can be read using any flowise powered document store or document reading function, then you can retrieve context from this new index, more modifications and improvements may be introduced in future PRs, some commits are related to our personal CI/CD pipeline which can be ignored or re-modified after merge, kindly have a look and address any issues in the comments
@MohamedYasserOaf I've updated the code, and moved it into Vector Store folder, can you give this a test?
I pulled the branch and applied pnpm install , the code fails on the build process , can't exactly point out the issue , i can send you the logs @HenryHengZJ
I pulled the branch and applied pnpm install , the code fails on the build process , can't exactly point out the issue , i can send you the logs @HenryHengZJ
because this pulls in the main branch with 2.0 changes, you need to
1.) remove all the node_modules
2.) pnpm i
3.) pnpm clean
4.) pnpm build-force
@HenryHengZJ I removed all the node_modules and followed your instructions , the issue still persist , maybe because my main is not up to date ?
@HenryHengZJ my friend was able to run it too, I will check the dependencies from my side, you will find the documents upserted on your index but you will not get notified, this is a functionality i need to implement, maybe i can add it to this PR, you can try to chat with the flow to test in the meantime
no I dont see anything upserted on Meilisearch, and chatting with it always return nothing
I will lookup the issue after solving my build issue, we are already using our custom flowise image with meilisearch retriever in our production environment, providing great results, will check and come back to you
@HenryHengZJ Hello Henry, I resolved my dependency problems and re-installed everything, I rebased with the latest commits from the main repository, now the build step fails because of a component called Melvis, the same happens with my friend
https://github.com/FlowiseAI/Flowise/blob/7153093bfc8c30d7095ac747e9d6a43fbc804c5d/packages/components/nodes/vectorstores/Milvus/Milvus.ts#L250
the above line causes the problem
The code build when I apply @ts-ignore on this line
regardless of the poor answer and documents used (I run locally with no gpu and old processor), the component is up and running , and i also synced my forked and updated my branch
yeah I realized thhat as well and solved this in this PR
did you upsert the data on Flowise? or you are just using for retrieval?
yes I upserted using the flow, you tried doing so ? we use a UI to visualize our indexes، I can shoot you a small video or a series of screenshots of the process, we also added features like an option to delete an index, we added metadata as searchable attributes for appling various filters when retrieving.
ah okay because I was not able to do so when I try upserting, so I wonder did that work for you.
feel free to send me the screenshot/videos!
@HenryHengZJ Hello henry, the following screenshot shows the whole process,
This is a screenshot of our flow, the index used is "my-index-1" , test bool document store contains around 7 pdfs chunked using unstructured.io
These screenshots shows "my-index-1" having 0 documents
The following screenshot shows my flowise upsert window
this is my index after upserting
This is a screenshot of my chat window using my flow, you can see it has retrieved context from my upserted index
- You can try to visualize your index in a 3rd party tool or via API calls to ensure your index is properly upserted
- You want to make sure that you entered your admin api key, I encountered this at the beginning I was using only the search API key
- Try to watch the terminal for any errors occured during upsert process